← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #59223

[Bug 1645767] [NEW] nova evacuate fails when encrypted volume is connected to instance

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Description
===========
If instance has encrypted volume connected to it and mounted during the host failure, 'nova evacuate' would result in instance entering the error state.

Steps to reproduce
==================
1. Boot a VM
a) nova boot --image 2c60a713-bbba-4696-adff-c80a12cab7d8 --flavor 42 --nic net-id=eb6c7f6b-100f-41bb-9c5d-11975a2cdba6 --availability-zone nova:compute1 test
b) nova floating-ip-associate test 192.168.57.50

2. Create an encrypted volume
a) cinder type-create LUKS
b) cinder encryption-type-create --cipher aes-xts-plain64 --key_size 512 \                 
  --control_location front-end LUKS nova.volume.encryptors.luks.LuksEncryptor
c) openstack volume create --size 1 --type 58931c06-1431-45cf-9b4c-4e998594447d test-volume

3. Attach the volume to instance
a) openstack server add volume test d06b7b7c-641d-4086-a3b1-4eee898b5c2a

4. SSH to instance, create file system on volume and mount it
a) ssh cirros@192.168.57.50
b) sudo mkfs.ext3 /dev/vdb
c) sudo mount /dev/vdb /mnt
d) sudo touch /mtn/test

5. Kill the compute node (In my case it was powering of the compute1)

6. Try to evacuate instance using `nova evacuate`

Expected result
===============
Instance rebuilded on another host, with volume attached to it.

Actual result
=============
Instance entered the error state. Output from `nova show test`:

Unexpected error while running command.           
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf cryptsetup luksOpen --key-file=- /dev/sdc crypt-ip-192.168.57.2:3260-iscsi-iqn.2010-10.org.openstack:volume-d06b7b7c-641d-4086-a3b1-4eee898b5c2a-lun-1 
 Exit code: 2", "code": 500, "details": "  File \"/opt/stack/nova/nova/compute/manager.py\", line 204, in decorated_function                                        
     return function(self, context, *args, **kwargs)                                        
   File \"/opt/stack/nova/nova/compute/manager.py\", line 2703, in rebuild_instance                                        
     bdms, recreate, on_shared_storage, preserve_ephemeral)                                        
   File \"/opt/stack/nova/nova/compute/manager.py\", line 2747, in _do_rebuild_instance_with_claim                                        
     self._do_rebuild_instance(*args, **kwargs)                                        
   File \"/opt/stack/nova/nova/compute/manager.py\", line 2862, in _do_rebuild_instance                                        
     self._rebuild_default_impl(**kwargs)                                        
   File \"/opt/stack/nova/nova/compute/manager.py\", line 2626, in _rebuild_default_impl                                        
     block_device_info=new_block_device_info)                                        
   File \"/opt/stack/nova/nova/virt/libvirt/driver.py\", line 2622, in spawn                                        
     post_xml_callback=gen_confdrive)                                        
   File \"/opt/stack/nova/nova/virt/libvirt/driver.py\", line 4845, in _create_domain_and_network                                        
     encryptor.attach_volume(context, **encryption)                                        
   File \"/opt/stack/nova/nova/volume/encryptors/luks.py\", line 102, in attach_volume                                        
     self._open_volume(passphrase, **kwargs)                                        
   File \"/opt/stack/nova/nova/volume/encryptors/luks.py\", line 86, in _open_volume                                        
     run_as_root=True, check_exit_code=True)                                        
   File \"/opt/stack/nova/nova/utils.py\", line 295, in execute                                        
     return RootwrapProcessHelper().execute(*cmd, **kwargs)                                        
   File \"/opt/stack/nova/nova/utils.py\", line 178, in execute                                        
     return processutils.execute(*cmd, **kwargs)                                        
   File \"/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py\", line 394, in execute                                        
     cmd=sanitized_cmd)      

Environment
===========

* Multinode devstack on VMs running Ubuntu 16.04
* Networking: Neutron with OVS
* Cinder backend - LVM

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1645767

Title:
  nova evacuate fails when encrypted volume is connected to instance

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  If instance has encrypted volume connected to it and mounted during the host failure, 'nova evacuate' would result in instance entering the error state.

  Steps to reproduce
  ==================
  1. Boot a VM
  a) nova boot --image 2c60a713-bbba-4696-adff-c80a12cab7d8 --flavor 42 --nic net-id=eb6c7f6b-100f-41bb-9c5d-11975a2cdba6 --availability-zone nova:compute1 test
  b) nova floating-ip-associate test 192.168.57.50

  2. Create an encrypted volume
  a) cinder type-create LUKS
  b) cinder encryption-type-create --cipher aes-xts-plain64 --key_size 512 \                 
    --control_location front-end LUKS nova.volume.encryptors.luks.LuksEncryptor
  c) openstack volume create --size 1 --type 58931c06-1431-45cf-9b4c-4e998594447d test-volume

  3. Attach the volume to instance
  a) openstack server add volume test d06b7b7c-641d-4086-a3b1-4eee898b5c2a

  4. SSH to instance, create file system on volume and mount it
  a) ssh cirros@192.168.57.50
  b) sudo mkfs.ext3 /dev/vdb
  c) sudo mount /dev/vdb /mnt
  d) sudo touch /mtn/test

  5. Kill the compute node (In my case it was powering of the compute1)

  6. Try to evacuate instance using `nova evacuate`

  Expected result
  ===============
  Instance rebuilded on another host, with volume attached to it.

  Actual result
  =============
  Instance entered the error state. Output from `nova show test`:

  Unexpected error while running command.           
  Command: sudo nova-rootwrap /etc/nova/rootwrap.conf cryptsetup luksOpen --key-file=- /dev/sdc crypt-ip-192.168.57.2:3260-iscsi-iqn.2010-10.org.openstack:volume-d06b7b7c-641d-4086-a3b1-4eee898b5c2a-lun-1 
   Exit code: 2", "code": 500, "details": "  File \"/opt/stack/nova/nova/compute/manager.py\", line 204, in decorated_function                                        
       return function(self, context, *args, **kwargs)                                        
     File \"/opt/stack/nova/nova/compute/manager.py\", line 2703, in rebuild_instance                                        
       bdms, recreate, on_shared_storage, preserve_ephemeral)                                        
     File \"/opt/stack/nova/nova/compute/manager.py\", line 2747, in _do_rebuild_instance_with_claim                                        
       self._do_rebuild_instance(*args, **kwargs)                                        
     File \"/opt/stack/nova/nova/compute/manager.py\", line 2862, in _do_rebuild_instance                                        
       self._rebuild_default_impl(**kwargs)                                        
     File \"/opt/stack/nova/nova/compute/manager.py\", line 2626, in _rebuild_default_impl                                        
       block_device_info=new_block_device_info)                                        
     File \"/opt/stack/nova/nova/virt/libvirt/driver.py\", line 2622, in spawn                                        
       post_xml_callback=gen_confdrive)                                        
     File \"/opt/stack/nova/nova/virt/libvirt/driver.py\", line 4845, in _create_domain_and_network                                        
       encryptor.attach_volume(context, **encryption)                                        
     File \"/opt/stack/nova/nova/volume/encryptors/luks.py\", line 102, in attach_volume                                        
       self._open_volume(passphrase, **kwargs)                                        
     File \"/opt/stack/nova/nova/volume/encryptors/luks.py\", line 86, in _open_volume                                        
       run_as_root=True, check_exit_code=True)                                        
     File \"/opt/stack/nova/nova/utils.py\", line 295, in execute                                        
       return RootwrapProcessHelper().execute(*cmd, **kwargs)                                        
     File \"/opt/stack/nova/nova/utils.py\", line 178, in execute                                        
       return processutils.execute(*cmd, **kwargs)                                        
     File \"/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py\", line 394, in execute                                        
       cmd=sanitized_cmd)      

  Environment
  ===========

  * Multinode devstack on VMs running Ubuntu 16.04
  * Networking: Neutron with OVS
  * Cinder backend - LVM

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1645767/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next