yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1691831] [NEW] VM evacuation is broken with shared torage if VM console.log is not owned by nova

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Cedric Brandily <1691831@xxxxxxxxxxxxxxxxxx>
Date: Thu, 18 May 2017 19:25:42 -0000
Reply-to: Bug 1691831 <1691831@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

On my Ocata deployment (with a shared storage between my KVMs hypervisors), the following worflow is failing:
 * stop nova-compute on a KVM hypervisor
 * stop a VM on the KVM hypervisor using virsh destroy
 * evacuate the VM ... which fails with the stacktrace:

ERROR nova.compute.manager [req-dcb547e3-5f98-488e-8dbf-ad4453ce82ac 7e6f47b9c6cf4994bb38a2eb3ad6243f 920a4480938349eca2651c140ce33fdd - - -] [instance: a129ab8f-c224-4df2-8134-6716cfe89acf] Setting instance vm_state to ERROR
ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf] Traceback (most recent call last):
ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6717, in _error_out_instance_on_exception
ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]     yield
ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2751, in rebuild_instance
ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]     bdms, recreate, on_shared_storage, preserve_ephemeral)
ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2795, in _do_rebuild_instance_with_claim
ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]     self._do_rebuild_instance(*args, **kwargs)
ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2910, in _do_rebuild_instance
ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]     self._rebuild_default_impl(**kwargs)
ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2673, in _rebuild_default_impl
ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]     block_device_info=new_block_device_info)
ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2689, in spawn
ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]     self._ensure_console_log_for_instance(instance)
ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2961, in _ensure_console_log_for_instance
ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]     libvirt_utils.file_open(console_file, 'a').close()
ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 350, in file_open
ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]     return open(*args, **kwargs)
ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf] IOError: [Errno 13] Permission denied: '/var/lib/nova/instances/a129ab8f-c224-4df2-8134-6716cfe89acf/console.log'
ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]


After some investigation:

_ensure_console_log_for_instance[1] ensures console.log existence. A
change[2] updated this method in order to succeed if the file exists
without nova being able to open it by ignoring EPERM erros (errno 1,
"operation not permitted") but it should ignore EACCES errors (errno 13,
"permission denied").


 >>> open('/etc/shadow')
 Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
 IOError: [Errno 13] Permission denied: '/etc/shadow'


EACCES errors are raised when you cannot do something because of insufficient permissions, EPERM are raised when you cannot do something (even with root account).


[1] nova.virt.libvirt.driver
[2] https://review.openstack.org/392643

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1691831

Title:
  VM evacuation is broken with shared torage if VM console.log is not
  owned by nova

Status in OpenStack Compute (nova):
  New

Bug description:
  On my Ocata deployment (with a shared storage between my KVMs hypervisors), the following worflow is failing:
   * stop nova-compute on a KVM hypervisor
   * stop a VM on the KVM hypervisor using virsh destroy
   * evacuate the VM ... which fails with the stacktrace:

  ERROR nova.compute.manager [req-dcb547e3-5f98-488e-8dbf-ad4453ce82ac 7e6f47b9c6cf4994bb38a2eb3ad6243f 920a4480938349eca2651c140ce33fdd - - -] [instance: a129ab8f-c224-4df2-8134-6716cfe89acf] Setting instance vm_state to ERROR
  ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf] Traceback (most recent call last):
  ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6717, in _error_out_instance_on_exception
  ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]     yield
  ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2751, in rebuild_instance
  ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]     bdms, recreate, on_shared_storage, preserve_ephemeral)
  ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2795, in _do_rebuild_instance_with_claim
  ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]     self._do_rebuild_instance(*args, **kwargs)
  ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2910, in _do_rebuild_instance
  ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]     self._rebuild_default_impl(**kwargs)
  ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2673, in _rebuild_default_impl
  ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]     block_device_info=new_block_device_info)
  ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2689, in spawn
  ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]     self._ensure_console_log_for_instance(instance)
  ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2961, in _ensure_console_log_for_instance
  ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]     libvirt_utils.file_open(console_file, 'a').close()
  ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 350, in file_open
  ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]     return open(*args, **kwargs)
  ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf] IOError: [Errno 13] Permission denied: '/var/lib/nova/instances/a129ab8f-c224-4df2-8134-6716cfe89acf/console.log'
  ERROR nova.compute.manager [instance: a129ab8f-c224-4df2-8134-6716cfe89acf]

  
  After some investigation:

  _ensure_console_log_for_instance[1] ensures console.log existence. A
  change[2] updated this method in order to succeed if the file exists
  without nova being able to open it by ignoring EPERM erros (errno 1,
  "operation not permitted") but it should ignore EACCES errors (errno
  13, "permission denied").

  
   >>> open('/etc/shadow')
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
   IOError: [Errno 13] Permission denied: '/etc/shadow'

  
  EACCES errors are raised when you cannot do something because of insufficient permissions, EPERM are raised when you cannot do something (even with root account).

  
  [1] nova.virt.libvirt.driver
  [2] https://review.openstack.org/392643

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1691831/+subscriptions
Follow ups

[Bug 1691831] Re: VM evacuation is broken with shared torage if VM console.log is not owned by nova
From: Matt Riedemann, 2017-05-30
[Bug 1691831] Re: VM evacuation is broken with shared torage if VM console.log is not owned by nova
From: OpenStack Infra, 2017-05-29