yahoo-eng-team team mailing list archive

[Ramy Asselin <ramy.asselin@xxxxxx>]

** Changed in: nova
       Status: Expired => Confirmed

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1439861

Title:
  encrypted iSCSI volume attach fails to attach when multipath-tools
  installed

Status in OpenStack Compute (Nova):
  Confirmed

Bug description:
  An error was occurring in a devstack setup with nova version:

  commit ab25f5f34b6ee37e495aa338aeb90b914f622b9d
  Merge "instance termination with update_dns_entries set fails"

  A volume-type encrypted with CryptsetupEncryptor was being used.  A
  volume was created using this volume-type and an attempt to attach it
  to an instance was made.  This error also occurred when using the
  LuksEncryptor for the volume-type.

  The following error occurred in n-cpu during attachment:

  Stack Trace:

  2015-04-02 13:39:54.397 ERROR nova.virt.block_device [req-a8220e7d-8d1e-459d-be1f-4ddd65b7ff66 admin admin] [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] Driver failed to attach volume 81c5f69a-9b01-4f
  c0-a105-be9d3c966aaf at /dev/vdb
  2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] Traceback (most recent call last):
  2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5]   File "/opt/stack/nova/nova/virt/block_device.py", line 251, in attach
  2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5]     device_type=self['device_type'], encryption=encryption)
  2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1064, in attach_volume
  2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5]     self._disconnect_volume(connection_info, disk_dev)
  2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5]     six.reraise(self.type_, self.value, self.tb)
  2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1051, in attach_volume
  2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5]     encryptor.attach_volume(context, **encryption)
  2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5]   File "/opt/stack/nova/nova/volume/encryptors/cryptsetup.py", line 93, in attach_volume
  2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5]     self._open_volume(passphrase, **kwargs)
  2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5]   File "/opt/stack/nova/nova/volume/encryptors/cryptsetup.py", line 78, in _open_volume
  2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5]     check_exit_code=True, run_as_root=True)
  2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5]   File "/opt/stack/nova/nova/utils.py", line 206, in execute
  2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5]     return processutils.execute(*cmd, **kwargs)
  2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5]   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 233, in execute
  2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5]     cmd=sanitized_cmd)
  2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] ProcessExecutionError: Unexpected error while running command.
  2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf cryptsetup create --key-file=- ip-10.50.3.20:3260-
  iscsi-iqn.2003-10.com.lefthandnetworks:vsa-12-721:853:volume-81c5f69a-9b01-4fc0-a105-be9d3c966aaf-lun-0 /dev/sdb
  2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] Exit code: 5
  2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] Stdout: u''
  2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] Stderr: u'Cannot use device /dev/sdb which is in use (already mapped or mounted).\n'
  2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5]

  After looking into reasons why the device would be busy it was found
  that uninstalling multipath-tools seemed to fix the problem.
  Multipath-tools appears to keep the device busy even if not used.

  iscsi_use_multipath was not set to True in nova.conf.

  Steps to reproduce:

  install multipath-tools and wait for multipath daemon service to start.
  attempt to attach an encrypted volume to an instance

  The above error should occur.

  uninstall multipath-tools and try to attach the volume again and it
  should work properly.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1439861/+subscriptions