yahoo-eng-team team mailing list archive

[OpenStack Infra <1997352@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/nova/+/906380
Committed: https://opendev.org/openstack/nova/commit/406d590a364d2c3ebc91e5f28f94011b158459d2
Submitter: "Zuul (22348)"
Branch:    master

commit 406d590a364d2c3ebc91e5f28f94011b158459d2
Author: Simon Hensel <simon.hensel@xxxxxxxxx>
Date:   Tue Jan 23 16:16:17 2024 +0100

    Always delete NVRAM files when deleting instances
    
    When deleting an instance, always send VIR_DOMAIN_UNDEFINE_NVRAM to
    delete the NVRAM file, regardless of whether the image is of type UEFI.
    This prevents a bug when rebuilding an instance from an UEFI image to a
    non-UEFI image.
    
    Closes-Bug: #1997352
    
    Change-Id: I24648f5b7895bf5d093f222b6c6e364becbb531f
    Signed-off-by: Simon Hensel <simon.hensel@xxxxxxxxx>


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1997352

Title:
  When REBUILDING from UEFI to non-UEFI instance ends up in ERROR state

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  If an UEFI instance is REBUILDED using a non-UEFI image as a
  replacement via e.g.:

  # openstack server create --flavor c4.2xlarge --image
  ubuntu-22.04-x86_64-uefi --network mynetwork --key-name mykey ubuntu-
  uefi-test --security-group default

  # openstack server rebuild --image ubuntu-22.04-x86_64 ubuntu-uefi-
  test

  
  The instance ends up in an error state:

  ```
  Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/nova/virt/libvirt/guest.py", line 285, in delete_configuration
      self._domain.undefineFlags(flags)
    File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 193, in doit
      result = proxy_call(self._autowrap, f, *args, **kwargs)
    File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 151, in proxy_call
      rv = execute(f, *args, **kwargs)
    File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 132, in execute
      six.reraise(c, e, tb)
    File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
      raise value
    File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 86, in tworker
      rv = meth(*args, **kwargs)
    File "/usr/lib/python3/dist-packages/libvirt.py", line 2924, in undefineFlags 
      if ret == -1: raise libvirtError (\'virDomainUndefineFlags() failed\', dom=self)
  libvirt.libvirtError: Requested operation is not valid: cannot undefine domain with nvram

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 200, in decorated_function
      return function(self, context, *args, **kwargs)
    File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 3095, in terminate_instance
      do_terminate_instance(instance, bdms)
    File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 360, in inner
      return f(*args, **kwargs)
    File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 3093, in do_terminate_instance
      self._set_instance_obj_error_state(instance)
    File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
      self.force_reraise()
    File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
      raise self.value
    File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 3083, in do_terminate_instance
      self._delete_instance(context, instance, bdms)
    File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 3018, in _delete_instance
      self._shutdown_instance(context, instance, bdms)
    File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2910, in _shutdown_instance
      self._try_deallocate_network(context, instance,
    File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
      self.force_reraise()
    File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
      raise self.value
   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2897, in _shutdown_instance
      self.driver.destroy(context, instance, network_info,
    File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1423, in destroy
      self.cleanup(context, instance, network_info, block_device_info,
    File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1493, in cleanup
      return self._cleanup(
    File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1585, in _cleanup
      self._undefine_domain(instance)
    File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1442, in _undefine_domain
      LOG.error(\'Error from libvirt during undefine. \'
    File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
      self.force_reraise()
    File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
      raise self.value
    File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1433, in _undefine_domain
      guest.delete_configuration(support_uefi)
    File "/usr/lib/python3/dist-packages/nova/virt/libvirt/guest.py", line 289, in delete_configuration
      self._domain.undefine()
    File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 193, in doit
      result = proxy_call(self._autowrap, f, *args, **kwargs)
    File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 151, in proxy_call
      rv = execute(f, *args, **kwargs)
    File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 132, in execute
      six.reraise(c, e, tb)
    File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
      raise value
    File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 86, in tworker
      rv = meth(*args, **kwargs)
    File "/usr/lib/python3/dist-packages/libvirt.py", line 2888, in undefine
      if ret == -1: raise libvirtError (\'virDomainUndefine() failed\', dom=self)
  libvirt.libvirtError: Requested operation is not valid: cannot undefine domain with nvram

  ```

  Additionally the instance cannot be deleted. Only manually by issuing
  a ``virsh undefine [instance_uuid] --nvram``

  
  I am pretty certain this is related to https://github.com/openstack/nova/commit/539d381434ccadcdc3f5d58c2705c35558a3a065 which introduced the deletion of nvram if the machine supports UEFI. Likely this info is lost on the switch to the non-UEFI image and therefore this flag is not set.

  
  I am wondering why this flag is not send to libvirt by default to always be able to delete instances UEFI or not?

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1997352/+subscriptions