← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #34927

[Bug 1470562] [NEW] 'in use' error when Nova volume encryptors format cinder volumes

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Tempest scenario TestEncryptedCinderVolumes has been silently skipped when run with NetApp iSCSI cinder volume drivers because they did not set the 'encrypted' key in the connection_info['data'] dict in their initialize_connection methods. Change
https://review.openstack.org/#/c/193673/ - which sets the encrypted flag generically, in the VolumeManager's initialize_connection, on the basis of the volume.encryption_key_id value - causes this test to actually run its encryption providers and exposes a problem in LuksEncryptor:_format_volume() for these iSCSI volumes.

In TestEncryptedCinderVolumes we get the following exception:

2015-06-29 06:27:18.866 ERROR nova.virt.libvirt.driver [req-7124728f-64b7-4951-806b-10901bb2f6b9 TestEncryptedCinderVolumes-1766245790 TestEncryptedCinderVolumes-1100414243] [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c] Failed to attach volume at mountpoint: /dev/vdb
2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c] Traceback (most recent call last):
2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1082, in attach_volume
2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c]     encryptor.attach_volume(context, **encryption)
2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c]   File "/opt/stack/new/nova/nova/volume/encryptors/luks.py", line 113, in attach_volume
2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c]     self._format_volume(passphrase, **kwargs)
2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c]   File "/opt/stack/new/nova/nova/volume/encryptors/luks.py", line 78, in _format_volume
2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c]     check_exit_code=True, run_as_root=True)
2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c]   File "/opt/stack/new/nova/nova/utils.py", line 229, in execute
2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c]     return processutils.execute(*cmd, **kwargs)
2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c]   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 260, in execute
2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c]     cmd=sanitized_cmd)
2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c] ProcessExecutionError: Unexpected error while running command.
2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf cryptsetup --batch-mode luksFormat --key-file=- --cipher aes-xts-plain64 --key-size 512 /dev/sdh
2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c] Exit code: 5
2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c] Stdout: u''
2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c] Stderr: u'Cannot format device /dev/sdh which is still in use.\n'
2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c] 

This is on master, corresponding code is:
https://github.com/openstack/nova/blob/master/nova/volume/encryptors/luks.py#L78

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1470562

Title:
  'in use' error when Nova volume encryptors format cinder volumes

Status in OpenStack Compute (Nova):
  New

Bug description:
  Tempest scenario TestEncryptedCinderVolumes has been silently skipped when run with NetApp iSCSI cinder volume drivers because they did not set the 'encrypted' key in the connection_info['data'] dict in their initialize_connection methods. Change
  https://review.openstack.org/#/c/193673/ - which sets the encrypted flag generically, in the VolumeManager's initialize_connection, on the basis of the volume.encryption_key_id value - causes this test to actually run its encryption providers and exposes a problem in LuksEncryptor:_format_volume() for these iSCSI volumes.

  In TestEncryptedCinderVolumes we get the following exception:

  2015-06-29 06:27:18.866 ERROR nova.virt.libvirt.driver [req-7124728f-64b7-4951-806b-10901bb2f6b9 TestEncryptedCinderVolumes-1766245790 TestEncryptedCinderVolumes-1100414243] [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c] Failed to attach volume at mountpoint: /dev/vdb
  2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c] Traceback (most recent call last):
  2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1082, in attach_volume
  2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c]     encryptor.attach_volume(context, **encryption)
  2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c]   File "/opt/stack/new/nova/nova/volume/encryptors/luks.py", line 113, in attach_volume
  2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c]     self._format_volume(passphrase, **kwargs)
  2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c]   File "/opt/stack/new/nova/nova/volume/encryptors/luks.py", line 78, in _format_volume
  2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c]     check_exit_code=True, run_as_root=True)
  2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c]   File "/opt/stack/new/nova/nova/utils.py", line 229, in execute
  2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c]     return processutils.execute(*cmd, **kwargs)
  2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c]   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 260, in execute
  2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c]     cmd=sanitized_cmd)
  2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c] ProcessExecutionError: Unexpected error while running command.
  2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf cryptsetup --batch-mode luksFormat --key-file=- --cipher aes-xts-plain64 --key-size 512 /dev/sdh
  2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c] Exit code: 5
  2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c] Stdout: u''
  2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c] Stderr: u'Cannot format device /dev/sdh which is still in use.\n'
  2015-06-29 06:27:18.866 12852 ERROR nova.virt.libvirt.driver [instance: 4a5dd4fd-78f8-42ea-8736-406ccb178d7c] 

  This is on master, corresponding code is:
  https://github.com/openstack/nova/blob/master/nova/volume/encryptors/luks.py#L78

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1470562/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next