← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #90442

[Bug 1997352] [NEW] When REBUILDING from UEFI to non-UEFI instance ends up in ERROR state

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

If an UEFI instance is REBUILDED using a non-UEFI image as a replacement
via e.g.:

# openstack server create --flavor c4.2xlarge --image
ubuntu-22.04-x86_64-uefi --network mynetwork --key-name mykey ubuntu-
uefi-test --security-group default

# openstack server rebuild --image ubuntu-22.04-x86_64 ubuntu-uefi-test


The instance ends up in an error state:

```
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/nova/virt/libvirt/guest.py", line 285, in delete_configuration
    self._domain.undefineFlags(flags)
  File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 193, in doit
    result = proxy_call(self._autowrap, f, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 151, in proxy_call
    rv = execute(f, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 132, in execute
    six.reraise(c, e, tb)
  File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
    raise value
  File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 86, in tworker
    rv = meth(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/libvirt.py", line 2924, in undefineFlags 
    if ret == -1: raise libvirtError (\'virDomainUndefineFlags() failed\', dom=self)
libvirt.libvirtError: Requested operation is not valid: cannot undefine domain with nvram

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 200, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 3095, in terminate_instance
    do_terminate_instance(instance, bdms)
  File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 360, in inner
    return f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 3093, in do_terminate_instance
    self._set_instance_obj_error_state(instance)
  File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()
  File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value
  File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 3083, in do_terminate_instance
    self._delete_instance(context, instance, bdms)
  File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 3018, in _delete_instance
    self._shutdown_instance(context, instance, bdms)
  File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2910, in _shutdown_instance
    self._try_deallocate_network(context, instance,
  File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()
  File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value
 File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2897, in _shutdown_instance
    self.driver.destroy(context, instance, network_info,
  File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1423, in destroy
    self.cleanup(context, instance, network_info, block_device_info,
  File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1493, in cleanup
    return self._cleanup(
  File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1585, in _cleanup
    self._undefine_domain(instance)
  File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1442, in _undefine_domain
    LOG.error(\'Error from libvirt during undefine. \'
  File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()
  File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value
  File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1433, in _undefine_domain
    guest.delete_configuration(support_uefi)
  File "/usr/lib/python3/dist-packages/nova/virt/libvirt/guest.py", line 289, in delete_configuration
    self._domain.undefine()
  File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 193, in doit
    result = proxy_call(self._autowrap, f, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 151, in proxy_call
    rv = execute(f, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 132, in execute
    six.reraise(c, e, tb)
  File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
    raise value
  File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 86, in tworker
    rv = meth(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/libvirt.py", line 2888, in undefine
    if ret == -1: raise libvirtError (\'virDomainUndefine() failed\', dom=self)
libvirt.libvirtError: Requested operation is not valid: cannot undefine domain with nvram

```

Additionally the instance cannot be deleted. Only manually by issuing a
``virsh undefine [instance_uuid] --nvram``


I am pretty certain this is related to https://github.com/openstack/nova/commit/539d381434ccadcdc3f5d58c2705c35558a3a065 which introduced the deletion of nvram if the machine supports UEFI. Likely this info is lost on the switch to the non-UEFI image and therefore this flag is not set.


I am wondering why this flag is not send to libvirt by default to always be able to delete instances UEFI or not?

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1997352

Title:
  When REBUILDING from UEFI to non-UEFI instance ends up in ERROR state

Status in OpenStack Compute (nova):
  New

Bug description:
  If an UEFI instance is REBUILDED using a non-UEFI image as a
  replacement via e.g.:

  # openstack server create --flavor c4.2xlarge --image
  ubuntu-22.04-x86_64-uefi --network mynetwork --key-name mykey ubuntu-
  uefi-test --security-group default

  # openstack server rebuild --image ubuntu-22.04-x86_64 ubuntu-uefi-
  test

  
  The instance ends up in an error state:

  ```
  Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/nova/virt/libvirt/guest.py", line 285, in delete_configuration
      self._domain.undefineFlags(flags)
    File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 193, in doit
      result = proxy_call(self._autowrap, f, *args, **kwargs)
    File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 151, in proxy_call
      rv = execute(f, *args, **kwargs)
    File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 132, in execute
      six.reraise(c, e, tb)
    File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
      raise value
    File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 86, in tworker
      rv = meth(*args, **kwargs)
    File "/usr/lib/python3/dist-packages/libvirt.py", line 2924, in undefineFlags 
      if ret == -1: raise libvirtError (\'virDomainUndefineFlags() failed\', dom=self)
  libvirt.libvirtError: Requested operation is not valid: cannot undefine domain with nvram

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 200, in decorated_function
      return function(self, context, *args, **kwargs)
    File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 3095, in terminate_instance
      do_terminate_instance(instance, bdms)
    File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 360, in inner
      return f(*args, **kwargs)
    File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 3093, in do_terminate_instance
      self._set_instance_obj_error_state(instance)
    File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
      self.force_reraise()
    File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
      raise self.value
    File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 3083, in do_terminate_instance
      self._delete_instance(context, instance, bdms)
    File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 3018, in _delete_instance
      self._shutdown_instance(context, instance, bdms)
    File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2910, in _shutdown_instance
      self._try_deallocate_network(context, instance,
    File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
      self.force_reraise()
    File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
      raise self.value
   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2897, in _shutdown_instance
      self.driver.destroy(context, instance, network_info,
    File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1423, in destroy
      self.cleanup(context, instance, network_info, block_device_info,
    File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1493, in cleanup
      return self._cleanup(
    File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1585, in _cleanup
      self._undefine_domain(instance)
    File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1442, in _undefine_domain
      LOG.error(\'Error from libvirt during undefine. \'
    File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
      self.force_reraise()
    File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
      raise self.value
    File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1433, in _undefine_domain
      guest.delete_configuration(support_uefi)
    File "/usr/lib/python3/dist-packages/nova/virt/libvirt/guest.py", line 289, in delete_configuration
      self._domain.undefine()
    File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 193, in doit
      result = proxy_call(self._autowrap, f, *args, **kwargs)
    File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 151, in proxy_call
      rv = execute(f, *args, **kwargs)
    File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 132, in execute
      six.reraise(c, e, tb)
    File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
      raise value
    File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 86, in tworker
      rv = meth(*args, **kwargs)
    File "/usr/lib/python3/dist-packages/libvirt.py", line 2888, in undefine
      if ret == -1: raise libvirtError (\'virDomainUndefine() failed\', dom=self)
  libvirt.libvirtError: Requested operation is not valid: cannot undefine domain with nvram

  ```

  Additionally the instance cannot be deleted. Only manually by issuing
  a ``virsh undefine [instance_uuid] --nvram``

  
  I am pretty certain this is related to https://github.com/openstack/nova/commit/539d381434ccadcdc3f5d58c2705c35558a3a065 which introduced the deletion of nvram if the machine supports UEFI. Likely this info is lost on the switch to the non-UEFI image and therefore this flag is not set.

  
  I am wondering why this flag is not send to libvirt by default to always be able to delete instances UEFI or not?

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1997352/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next