yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1634282] [NEW] Nova fails to open console.log file at repeated host evacuation

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Marian Krcmarik <mkrcmari@xxxxxxxxxx>
Date: Mon, 17 Oct 2016 21:55:40 -0000
Reply-to: Bug 1634282 <1634282@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I am trying to perform "ping-pong" host evacuation - take down ungracefully host A, evacuate host A to host B, take down ungracefull host B and evacuate to host B. The second evacuation fails at following error:
[Errno 13] Permission denied: '/var/lib/nova/instances/e17a325c-f0da-4f2f-aad3-4b1c098f295f/console.log'

Traceback:
2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9] Traceback (most recent call last):
2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6566, in _error_out_instance_on_exception
2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9]     yield
2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2687, in rebuild_instance
2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9]     bdms, recreate, on_shared_storage, preserve_ephemeral)
2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2731, in _do_rebuild_instance_with_claim
2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9]     self._do_rebuild_instance(*args, **kwargs)
2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2846, in _do_rebuild_instance
2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9]     self._rebuild_default_impl(**kwargs)
2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2612, in _rebuild_default_impl
2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9]     block_device_info=new_block_device_info)
2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2574, in spawn
2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9]     self._ensure_console_log_for_instance(instance)
2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2835, in _ensure_console_log_for_instance
2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9]     libvirt_utils.file_open(console_file, 'a').close()
2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 313, in file_open
2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9]     return open(*args, **kwargs)
2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9] IOError: [Errno 13] Permission denied: '/var/lib/nova/instances/a7ba9743-b425-4e47-aeb0-d545e66fffe9/console.log'

The problem is that the console file is left on host A with qemu:qemu ownership set by libvirt after first evacuation (the host A was taken down ungracefully) and once nova tries to open that file with append permission under nova user during second evacution from host B to host A It fails due to permissions. This try of nova to open the console.log file was introduce in:
https://github.com/openstack/nova/commit/ec6ed24cb844dcdf834d283d496c9b920ff1db83

Since I believe that in default installations usually
dynamic_ownership=0 is not set and qemu is not started by the same user
as Nova I would consider this as a regression.

Steps to reproduce:
1. Boot a VM on host A
2. Disrupt host A and trigger evacuation to host B
3. Wait for host A to be online again
4. Disrupt host B and trigger evacuation to host A

Expected result:
Successful 2nd host evacuation

Actual result:
Nova does not have permission to open console.log file and fails the evacuation.

Newton release used

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1634282

Title:
  Nova fails to open console.log file at repeated host evacuation

Status in OpenStack Compute (nova):
  New

Bug description:
  I am trying to perform "ping-pong" host evacuation - take down ungracefully host A, evacuate host A to host B, take down ungracefull host B and evacuate to host B. The second evacuation fails at following error:
  [Errno 13] Permission denied: '/var/lib/nova/instances/e17a325c-f0da-4f2f-aad3-4b1c098f295f/console.log'

  Traceback:
  2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9] Traceback (most recent call last):
  2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6566, in _error_out_instance_on_exception
  2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9]     yield
  2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2687, in rebuild_instance
  2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9]     bdms, recreate, on_shared_storage, preserve_ephemeral)
  2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2731, in _do_rebuild_instance_with_claim
  2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9]     self._do_rebuild_instance(*args, **kwargs)
  2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2846, in _do_rebuild_instance
  2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9]     self._rebuild_default_impl(**kwargs)
  2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2612, in _rebuild_default_impl
  2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9]     block_device_info=new_block_device_info)
  2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2574, in spawn
  2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9]     self._ensure_console_log_for_instance(instance)
  2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2835, in _ensure_console_log_for_instance
  2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9]     libvirt_utils.file_open(console_file, 'a').close()
  2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 313, in file_open
  2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9]     return open(*args, **kwargs)
  2016-10-17 11:36:56.023 3433 ERROR nova.compute.manager [instance: a7ba9743-b425-4e47-aeb0-d545e66fffe9] IOError: [Errno 13] Permission denied: '/var/lib/nova/instances/a7ba9743-b425-4e47-aeb0-d545e66fffe9/console.log'

  The problem is that the console file is left on host A with qemu:qemu ownership set by libvirt after first evacuation (the host A was taken down ungracefully) and once nova tries to open that file with append permission under nova user during second evacution from host B to host A It fails due to permissions. This try of nova to open the console.log file was introduce in:
  https://github.com/openstack/nova/commit/ec6ed24cb844dcdf834d283d496c9b920ff1db83

  Since I believe that in default installations usually
  dynamic_ownership=0 is not set and qemu is not started by the same
  user as Nova I would consider this as a regression.

  Steps to reproduce:
  1. Boot a VM on host A
  2. Disrupt host A and trigger evacuation to host B
  3. Wait for host A to be online again
  4. Disrupt host B and trigger evacuation to host A

  Expected result:
  Successful 2nd host evacuation

  Actual result:
  Nova does not have permission to open console.log file and fails the evacuation.

  Newton release used

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1634282/+subscriptions
Follow ups

[Bug 1634282] Re: Nova fails to open console.log file at repeated host evacuation
From: Stephen Finucane, 2017-06-26