yahoo-eng-team team mailing list archive

[Lee Yarwood <lyarwood@xxxxxxxxxx>]

** Also affects: os-brick
   Importance: Undecided
       Status: New

** No longer affects: os-brick

** Project changed: nova => os-brick

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1511255

Title:
  Data corrupted in cinder nfs volume with encrypted volume type  after
  detached

Status in os-brick:
  Confirmed

Bug description:
  Summary:
  It fails to re-attach an encrypted volume created from nfs to an instance.

  Env:
  All in one with devstack and origin/mater repository.

  Reproduce steps:
  1. Create an cinder volume from nfs with encrypted volume type.
  2. Attach this volume to an instance.
  3. Detach it.
  4. Re-attach this volume to the instance.

  Expect result:
  re-attach succeeds.

  Actual result:
  The command fails with following error mesage:
  ne 89, in _open_volume^M
  ^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m    run_as_root=True, check_exit_code=True)^M
  ^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m  File "/opt/stack/nova/nova/utils.py", line 389, in execute^M
  ^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m    return RootwrapProcessHelper().execute(*cmd, **kwargs)^M
  ^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m  File "/opt/stack/nova/nova/utils.py", line 272, in execute^M
  ^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m    return processutils.execute(*cmd, **kwargs)^M
  ^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m  File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 295, in execute^M
  ^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m    cmd=sanitized_cmd)^M
  ^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00mProcessExecutionError: Unexpected error while running command.^M
  ^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00mCommand: sudo nova-rootwrap /etc/nova/rootwrap.conf cryptsetup luksOpen --key-file=- /dev/mapper/volume-d55c2436-3453-47ef-977c-42ef2a334323 volume-d55c2436-3453-47ef-977c-42ef2a334323^M
  ^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00mExit code: 4^M
  ^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00mStdout: u''^M
  ^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00mStderr: u"Device /dev/mapper/volume-d55c2436-3453-47ef-977c-42ef2a334323 doesn't exist or access denied.\n"^M

  
  Analysis:
  When cinder creates a nfs volume, it is created in the share folder. For example, 
  /opt/stack/data/cinder/mnt/690e8d688d986e9d44a1a00ed99912d0/volume-d55c2436-3453-47ef-977c-42ef2a334323

  And when it is attached to an instance, the share folder is mounted to the compute node.
  /opt/stack/data/nova/mnt/690e8d688d986e9d44a1a00ed99912d0/volume-d55c2436-3453-47ef-977c-42ef2a334323

  -rw-rw-rw- 1 stack  stack   1073741824 Oct 29 07:52 volume-
  d55c2436-3453-47ef-977c-42ef2a334323

  As the volume type is encrypted, it needs to encryptors.attach_volume().
  With current encryptors mechanism, the file is changed to 
  lrwxrwxrwx 1 nobody nogroup         55 Oct 29 05:09 volume-d55c2436-3453-47ef-977c-42ef2a334323 -> /dev/mapper/volume-d55c2436-3453-47ef-977c-42ef2a334323

  It means the original cinder volume file is deleted, and it is a link
  pointed to the encrypted device.

  When detached, the encrypted device(/dev/mapper/volume-d55c2436-3453
  -47ef-977c-42ef2a334323) is deleted, and the above volume-
  d55c2436-3453-47ef-977c-42ef2a334323 in share folder is left as a file
  link.

  As a result, the volume is corrupted and re-attach fails

To manage notifications about this bug go to:
https://bugs.launchpad.net/os-brick/+bug/1511255/+subscriptions