yahoo-eng-team team mailing list archive

[OpenStack Infra <1905701@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/nova/+/793463
Committed: https://opendev.org/openstack/nova/commit/26d65fc882e42b824409dff87ff026dee1debe20
Submitter: "Zuul (22348)"
Branch:    master

commit 26d65fc882e42b824409dff87ff026dee1debe20
Author: Lee Yarwood <lyarwood@xxxxxxxxxx>
Date:   Thu May 27 16:47:26 2021 +0100

    libvirt: Do not destroy volume secrets during _hard_reboot
    
    Ia2007bc63ef09931ea0197cef29d6a5614ed821a unfortunately missed that
    resume_state_on_host_boot calls down into _hard_reboot always removing
    volume secrets rendering that change useless.
    
    This change seeks to address this by using the destroy_secrets kwarg
    introduced by I856268b371f7ba712b02189db3c927cd762a4dc3 within the
    _hard_reboot method of the libvirt driver to ensure secrets are not
    removed during a hard reboot.
    
    This resolves the original issue in bug #1905701 *and* allows admins to
    hard reboot a users instance when that instance has encrypted volumes
    attached with secrets stored in Barbican. This latter use case being
    something we can easily test within tempest unlike the compute reboot in
    bug #1905701.
    
    This change is kept small as it should ideally be backported alongside
    Ia2007bc63ef09931ea0197cef29d6a5614ed821a to stable/queens. Follow up
    changes on master will improve formatting, doc text and introduce
    functional tests to further validate this new behaviour of hard reboot
    within the libvirt driver.
    
    Closes-Bug: #1905701
    Change-Id: I3d1b21ba6eb3f5eb728693197c24b4b315eef821


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1905701

Title:
  Do not recreate libvirt secret when one already exists on the host
  during a host reboot

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) queens series:
  In Progress
Status in OpenStack Compute (nova) rocky series:
  In Progress
Status in OpenStack Compute (nova) stein series:
  In Progress
Status in OpenStack Compute (nova) train series:
  In Progress
Status in OpenStack Compute (nova) ussuri series:
  In Progress
Status in OpenStack Compute (nova) victoria series:
  Fix Released
Status in OpenStack Compute (nova) wallaby series:
  New
Status in OpenStack Compute (nova) xena series:
  Fix Released

Bug description:
  Description
  ===========

  When [compute]/resume_guests_state_on_host_boot is enabled the compute
  manager will attempt to restart instances on start up.

  When using the libvirt driver and instances with attached LUKSv1
  encrypted volumes a call is made to _attach_encryptor that currently
  assumes that any volume libvirt secrets don't already exist on the
  host. As a result this call will currently lead to an attempt to
  lookup encryption metadata that fails as the compute service is using
  a bare bones local only admin context to drive the restart of the
  instances.

  The libvirt secrets associated with LUKSv1 encrypted volumes actually
  persist a host reboot and thus this call to fetch encryption metadata,
  fetch the symmetric key etc are not required. Removal of these calls
  in this context should allow the compute service to start instances
  with these volumes attached.

  Steps to reproduce
  ==================
  * Enable [compute]/resume_guests_state_on_host_boot
  * Launch instances with encrypted LUKSv1 volumes attached
  * Reboot the underlying host

  Expected result
  ===============
  * The instances are restarted successfully by Nova as no external calls are made and the existing libvirt secret for any encrypted LUKSv1 volumes are reused.

  Actual result
  =============
  * The instances fail to restart as the initial calls made by the Nova service use an empty admin context without a service catelog etc.

  Environment
  ===========
  1. Exact version of OpenStack you are running. See the following

     master

  2. Which hypervisor did you use?
     (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
     What's the version of that?

     libvirt + QEMU/KVM

  2. Which storage type did you use?
     (For example: Ceph, LVM, GPFS, ...)
     What's the version of that?

     N/A

  3. Which networking type did you use?
     (For example: nova-network, Neutron with OpenVSwitch, ...)

     N/A

  Logs & Configs
  ==============

  2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf]   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1641, in _connect_volume
  2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf]     self._attach_encryptor(context, connection_info, encryption)
  2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf]   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1760, in _attach_encryptor
  2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf]     key = keymgr.get(context, encryption['encryption_key_id'])
  2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf]   File "/usr/lib/python3.6/site-packages/castellan/key_manager/barbican_key_manager.py", line 575, in get
  2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf]     secret = self._get_secret(context, managed_object_id)
  2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf]   File "/usr/lib/python3.6/site-packages/castellan/key_manager/barbican_key_manager.py", line 545, in _ge
  t_secret
  2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf]     barbican_client = self._get_barbican_client(context)
  2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf]   File "/usr/lib/python3.6/site-packages/castellan/key_manager/barbican_key_manager.py", line 142, in _ge
  t_barbican_client
  2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf]     self._barbican_endpoint)
  2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf]   File "/usr/lib/python3.6/site-packages/castellan/key_manager/barbican_key_manager.py", line 214, in _cr
  eate_base_url
  2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf]     service_type='key-manager')
  2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf]   File "/usr/lib/python3.6/site-packages/keystoneauth1/access/service_catalog.py", line 425, in endpoint_
  data_for
  2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf]     raise exceptions.EmptyCatalog('The service catalog is empty.')
  2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf] keystoneauth1.exceptions.catalog.EmptyCatalog: The service catalog is empty.
  2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1905701/+subscriptions