← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1797115] [NEW] Instance data deleted on shared storage after failed evacuate (queens)

 

Public bug reported:

Hi
The issue was experienced after one of compute nodes crashed. Then, after reboot of compute node, few instances was restarted, but rest of instances failed to start with error:
"Failed to get "write" lock
Is another process using the image?"

I tried to evacuate one instance to another host:
nova evacuate instance host2

After that nova compute on destination host throwed the same error:
Setting instance vm_state to ERROR: libvirtError: internal error: process exited while connecting to monitor: 2018-10-10T10:16:09.816477Z qemu-system-x86_64: -drive file=/var/lib/nova/instances/ed7760a8-3008-4feb-83f3-3b753b0e7d6e/disk,format=qcow2,if=none,id=drive-virtio-disk0,cache=none: Failed to get "write" lock
Is another process using the image?

Regardless of reason of this issue, the whole directory of the instance
was gone from shared nfs, so /var/lib/nova/instances/ed7760a8-3008-4feb-
83f3-3b753b0e7d6e was removed with instance disk.

Then I reproduced the error. After backup of disk file, evacuate and
restore instance directory with disk file, the instance was started
successfully.

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: evacuate

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1797115

Title:
  Instance data deleted on shared storage after failed evacuate (queens)

Status in OpenStack Compute (nova):
  New

Bug description:
  Hi
  The issue was experienced after one of compute nodes crashed. Then, after reboot of compute node, few instances was restarted, but rest of instances failed to start with error:
  "Failed to get "write" lock
  Is another process using the image?"

  I tried to evacuate one instance to another host:
  nova evacuate instance host2

  After that nova compute on destination host throwed the same error:
  Setting instance vm_state to ERROR: libvirtError: internal error: process exited while connecting to monitor: 2018-10-10T10:16:09.816477Z qemu-system-x86_64: -drive file=/var/lib/nova/instances/ed7760a8-3008-4feb-83f3-3b753b0e7d6e/disk,format=qcow2,if=none,id=drive-virtio-disk0,cache=none: Failed to get "write" lock
  Is another process using the image?

  Regardless of reason of this issue, the whole directory of the
  instance was gone from shared nfs, so
  /var/lib/nova/instances/ed7760a8-3008-4feb-83f3-3b753b0e7d6e was
  removed with instance disk.

  Then I reproduced the error. After backup of disk file, evacuate and
  restore instance directory with disk file, the instance was started
  successfully.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1797115/+subscriptions