yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1439855] Re: encrypted iSCSI volume fails to attach, name too long

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sean Dague <sean@xxxxxxxxx>
Date: Fri, 03 Apr 2015 15:17:32 -0000
Reply-to: Bug 1439855 <1439855@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
This isn't really a dup, because this is about nova failing gracefully I
think.

Can you provide relevant dmesg logs to figure out if this being
triggered in the kernel or if this is something in cryptsetup itself, so
that we can handle this in the right place?

** This bug is no longer a duplicate of bug 1432490
   TestEncryptedCinderVolumes cryptsetup name is too long

** Changed in: nova
       Status: New => Incomplete

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1439855

Title:
  encrypted iSCSI volume fails to attach, name too long

Status in OpenStack Compute (Nova):
  Incomplete

Bug description:
  When running the following tempest tests an error occurs in n-cpu:

  tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_cryptsetup
  test_encrypted_cinder_volumes_luks

  This occurred when using devstack with nova at:

  HEAD is now at d0c2684 Merge "libvirt: Resize down an instance booted
  from a volume"

  Both stack traces below are from n-cpu logs:

  Stack Trace
  (tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_cryptsetup):

  2015-03-27 18:03:07.990 ERROR nova.compute.manager [req-cc941973-c038-4bac-a5ca-d516cd5dd33d TestEncryptedCinderVolumes-658052177 TestEncryptedCinderVolumes-1476988517] [instance: a2cccacf-2876-4e94-94e0-dbb3fbf51c46] Failed to attach 2ab47be7-64ac-4d34-a38c-59c5e97e2ec2 at /dev/vdb
  2015-03-27 18:03:07.990 29082 TRACE nova.compute.manager [instance: a2cccacf-2876-4e94-94e0-dbb3fbf51c46] Traceback (most recent call last):
  2015-03-27 18:03:07.990 29082 TRACE nova.compute.manager [instance: a2cccacf-2876-4e94-94e0-dbb3fbf51c46]   File "/opt/stack/new/nova/nova/compute/manager.py", line 4735, in _attach_volume
  2015-03-27 18:03:07.990 29082 TRACE nova.compute.manager [instance: a2cccacf-2876-4e94-94e0-dbb3fbf51c46]     do_check_attach=False, do_driver_attach=True)
  2015-03-27 18:03:07.990 29082 TRACE nova.compute.manager [instance: a2cccacf-2876-4e94-94e0-dbb3fbf51c46]   File "/opt/stack/new/nova/nova/virt/block_device.py", line 48, in wrapped
  2015-03-27 18:03:07.990 29082 TRACE nova.compute.manager [instance: a2cccacf-2876-4e94-94e0-dbb3fbf51c46]     ret_val = method(obj, context, *args, **kwargs)
  2015-03-27 18:03:07.990 29082 TRACE nova.compute.manager [instance: a2cccacf-2876-4e94-94e0-dbb3fbf51c46]   File "/opt/stack/new/nova/nova/virt/block_device.py", line 260, in attach
  2015-03-27 18:03:07.990 29082 TRACE nova.compute.manager [instance: a2cccacf-2876-4e94-94e0-dbb3fbf51c46]     connector)
  2015-03-27 18:03:07.990 29082 TRACE nova.compute.manager [instance: a2cccacf-2876-4e94-94e0-dbb3fbf51c46]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2015-03-27 18:03:07.990 29082 TRACE nova.compute.manager [instance: a2cccacf-2876-4e94-94e0-dbb3fbf51c46]     six.reraise(self.type_, self.value, self.tb)
  2015-03-27 18:03:07.990 29082 TRACE nova.compute.manager [instance: a2cccacf-2876-4e94-94e0-dbb3fbf51c46]   File "/opt/stack/new/nova/nova/virt/block_device.py", line 251, in attach
  2015-03-27 18:03:07.990 29082 TRACE nova.compute.manager [instance: a2cccacf-2876-4e94-94e0-dbb3fbf51c46]     device_type=self['device_type'], encryption=encryption)
  2015-03-27 18:03:07.990 29082 TRACE nova.compute.manager [instance: a2cccacf-2876-4e94-94e0-dbb3fbf51c46]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1065, in attach_volume
  2015-03-27 18:03:07.990 29082 TRACE nova.compute.manager [instance: a2cccacf-2876-4e94-94e0-dbb3fbf51c46]     self._disconnect_volume(connection_info, disk_dev)
  2015-03-27 18:03:07.990 29082 TRACE nova.compute.manager [instance: a2cccacf-2876-4e94-94e0-dbb3fbf51c46]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2015-03-27 18:03:07.990 29082 TRACE nova.compute.manager [instance: a2cccacf-2876-4e94-94e0-dbb3fbf51c46]     six.reraise(self.type_, self.value, self.tb)
  2015-03-27 18:03:07.990 29082 TRACE nova.compute.manager [instance: a2cccacf-2876-4e94-94e0-dbb3fbf51c46]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1052, in attach_volume
  2015-03-27 18:03:07.990 29082 TRACE nova.compute.manager [instance: a2cccacf-2876-4e94-94e0-dbb3fbf51c46]     encryptor.attach_volume(context, **encryption)
  2015-03-27 18:03:07.990 29082 TRACE nova.compute.manager [instance: a2cccacf-2876-4e94-94e0-dbb3fbf51c46]   File "/opt/stack/new/nova/nova/volume/encryptors/cryptsetup.py", line 86, in attach_volume
  2015-03-27 18:03:07.990 29082 TRACE nova.compute.manager [instance: a2cccacf-2876-4e94-94e0-dbb3fbf51c46]     self._open_volume(passphrase, **kwargs)
  2015-03-27 18:03:07.990 29082 TRACE nova.compute.manager [instance: a2cccacf-2876-4e94-94e0-dbb3fbf51c46]   File "/opt/stack/new/nova/nova/volume/encryptors/cryptsetup.py", line 71, in _open_volume
  2015-03-27 18:03:07.990 29082 TRACE nova.compute.manager [instance: a2cccacf-2876-4e94-94e0-dbb3fbf51c46]     check_exit_code=True, run_as_root=True)
  2015-03-27 18:03:07.990 29082 TRACE nova.compute.manager [instance: a2cccacf-2876-4e94-94e0-dbb3fbf51c46]   File "/opt/stack/new/nova/nova/utils.py", line 206, in execute
  2015-03-27 18:03:07.990 29082 TRACE nova.compute.manager [instance: a2cccacf-2876-4e94-94e0-dbb3fbf51c46]     return processutils.execute(*cmd, **kwargs)
  2015-03-27 18:03:07.990 29082 TRACE nova.compute.manager [instance: a2cccacf-2876-4e94-94e0-dbb3fbf51c46]   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 233, in execute
  2015-03-27 18:03:07.990 29082 TRACE nova.compute.manager [instance: a2cccacf-2876-4e94-94e0-dbb3fbf51c46]     cmd=sanitized_cmd)
  2015-03-27 18:03:07.990 29082 TRACE nova.compute.manager [instance: a2cccacf-2876-4e94-94e0-dbb3fbf51c46] ProcessExecutionError: Unexpected error while running command.
  2015-03-27 18:03:07.990 29082 TRACE nova.compute.manager [instance: a2cccacf-2876-4e94-94e0-dbb3fbf51c46] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf cryptsetup create --key-file=- --cipher aes-xts-plain64 --key-size 512 ip-10.52.1.17:3260-iscsi-iqn.2003-10.com.lefthandnetworks:ci-vsa-12-725:159016:volume-2ab47be7-64ac-4d34-a38c-59c5e97e2ec2-lun-0 /dev/sdb
  2015-03-27 18:03:07.990 29082 TRACE nova.compute.manager [instance: a2cccacf-2876-4e94-94e0-dbb3fbf51c46] Exit code: 1
  2015-03-27 18:03:07.990 29082 TRACE nova.compute.manager [instance: a2cccacf-2876-4e94-94e0-dbb3fbf51c46] Stdout: u''
  2015-03-27 18:03:07.990 29082 TRACE nova.compute.manager [instance: a2cccacf-2876-4e94-94e0-dbb3fbf51c46] Stderr: u'Name "ip-10.52.1.17:3260-iscsi-iqn.2003-10.com.lefthandnetworks:ci-vsa-12-725:159016:volume-2ab47be7-64ac-4d34-a38c-59c5e97e2ec2-lun-0" too long.\n'
  2015-03-27 18:03:07.990 29082 TRACE nova.compute.manager [instance: a2cccacf-2876-4e94-94e0-dbb3fbf51c46]

  
  Stack Trace (test_encrypted_cinder_volumes_luks):

  2015-03-27 18:06:48.928 ERROR nova.virt.libvirt.driver [req-7167f5a7-c620-474a-a6e0-1cc992192930 TestEncryptedCinderVolumes-658052177 TestEncryptedCinderVolumes-1476988517] [instance: ddeda0aa-1457-41d2-82e9-9a4ae888125a] Failed to attach volume at mountpoint: /dev/vdb
  2015-03-27 18:06:48.928 29082 TRACE nova.virt.libvirt.driver [instance: ddeda0aa-1457-41d2-82e9-9a4ae888125a] Traceback (most recent call last):
  2015-03-27 18:06:48.928 29082 TRACE nova.virt.libvirt.driver [instance: ddeda0aa-1457-41d2-82e9-9a4ae888125a]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1052, in attach_volume
  2015-03-27 18:06:48.928 29082 TRACE nova.virt.libvirt.driver [instance: ddeda0aa-1457-41d2-82e9-9a4ae888125a]     encryptor.attach_volume(context, **encryption)
  2015-03-27 18:06:48.928 29082 TRACE nova.virt.libvirt.driver [instance: ddeda0aa-1457-41d2-82e9-9a4ae888125a]   File "/opt/stack/new/nova/nova/volume/encryptors/luks.py", line 114, in attach_volume
  2015-03-27 18:06:48.928 29082 TRACE nova.virt.libvirt.driver [instance: ddeda0aa-1457-41d2-82e9-9a4ae888125a]     self._open_volume(passphrase, **kwargs)
  2015-03-27 18:06:48.928 29082 TRACE nova.virt.libvirt.driver [instance: ddeda0aa-1457-41d2-82e9-9a4ae888125a]   File "/opt/stack/new/nova/nova/volume/encryptors/luks.py", line 89, in _open_volume
  2015-03-27 18:06:48.928 29082 TRACE nova.virt.libvirt.driver [instance: ddeda0aa-1457-41d2-82e9-9a4ae888125a]     run_as_root=True, check_exit_code=True)
  2015-03-27 18:06:48.928 29082 TRACE nova.virt.libvirt.driver [instance: ddeda0aa-1457-41d2-82e9-9a4ae888125a]   File "/opt/stack/new/nova/nova/utils.py", line 206, in execute
  2015-03-27 18:06:48.928 29082 TRACE nova.virt.libvirt.driver [instance: ddeda0aa-1457-41d2-82e9-9a4ae888125a]     return processutils.execute(*cmd, **kwargs)
  2015-03-27 18:06:48.928 29082 TRACE nova.virt.libvirt.driver [instance: ddeda0aa-1457-41d2-82e9-9a4ae888125a]   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 233, in execute
  2015-03-27 18:06:48.928 29082 TRACE nova.virt.libvirt.driver [instance: ddeda0aa-1457-41d2-82e9-9a4ae888125a]     cmd=sanitized_cmd)
  2015-03-27 18:06:48.928 29082 TRACE nova.virt.libvirt.driver [instance: ddeda0aa-1457-41d2-82e9-9a4ae888125a] ProcessExecutionError: Unexpected error while running command.
  2015-03-27 18:06:48.928 29082 TRACE nova.virt.libvirt.driver [instance: ddeda0aa-1457-41d2-82e9-9a4ae888125a] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf cryptsetup luksOpen --key-file=- /dev/sdb ip-10.52.1.17:3260-iscsi-iqn.2003-10.com.lefthandnetworks:ci-vsa-12-725:159021:volume-e74c95eb-e69b-4cf3-ac37-74e4ef38f796-lun-0
  2015-03-27 18:06:48.928 29082 TRACE nova.virt.libvirt.driver [instance: ddeda0aa-1457-41d2-82e9-9a4ae888125a] Exit code: 1
  2015-03-27 18:06:48.928 29082 TRACE nova.virt.libvirt.driver [instance: ddeda0aa-1457-41d2-82e9-9a4ae888125a] Stdout: u''
  2015-03-27 18:06:48.928 29082 TRACE nova.virt.libvirt.driver [instance: ddeda0aa-1457-41d2-82e9-9a4ae888125a] Stderr: u'Name "ip-10.52.1.17:3260-iscsi-iqn.2003-10.com.lefthandnetworks:ci-vsa-12-725:159021:volume-e74c95eb-e69b-4cf3-ac37-74e4ef38f796-lun-0" too long.\n'
  2015-03-27 18:06:48.928 29082 TRACE nova.virt.libvirt.driver [instance: ddeda0aa-1457-41d2-82e9-9a4ae888125a]

  
  To reproduce, run the tempest tests mentioned above on system that has an ip, iscsi properties and volume name which, when combined, results in a very long name used by cryptsetup.  The long name will cause the above error to occur.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1439855/+subscriptions
References

[Bug 1439855] [NEW] encrypted iSCSI volume fails to attach, name too long
From: Anthony Lee, 2015-04-02