yahoo-eng-team team mailing list archive

[OpenStack Infra <2002606@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/nova/+/872385
Committed: https://opendev.org/openstack/nova/commit/714894a7ff81e23f5ac7545a6b91ea484447a988
Submitter: "Zuul (22348)"
Branch:    master

commit 714894a7ff81e23f5ac7545a6b91ea484447a988
Author: Maxim Monin <maximmonin@xxxxxxxxx>
Date:   Wed Feb 1 08:33:18 2023 +0000

    Server Rescue leads to Server ERROR state if original image is deleted
    
    Server rescue in stable device mode has dependency on the original
    image used to create or rebuild the server. If the original image
    was deleted from Glance, the server could not be rescued. Nova has
    server.image_ref reference to non-existent image. Rescue in stable
    device rescue mode leads to driver error and Server in ERROR state.
    Proposed fix use falling back to the instance image metadata if
    the original image is not found in Glance.
    
    Closes-Bug: #2002606
    Change-Id: Ia0f747158721f82ef1803618b6cb30661b6252c9


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2002606

Title:
  Server Rescue leads to VM ERROR state if VM base image is deleted

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Server has dependency of original base image used to Create Server or
  Rebuild Server. If original image is deleted from glance, nova has
  server.image.id reference to non-existent image. Rescue in stable
  device rescue mode leads to driver error and Server in ERROR state.

  Step to reproduce:
  1. Create server from glance image.
  2. Delete original glance image.
  3. Run rescue server with input parameter of rescue image CD with hw_rescue_device and hw_rescue_bus image properties defined.
  Second use case:
  1. Restore Server from backup (with server Rebuild)
  2. Delete backup image.
  3. Run rescue in stable device mode.

  Result: Instance 73706a9a-e976-4024-8068-c439404ec953 cannot be rescued: Driver Error: Image be0e302d-79bd-44e1-93a5-68661e93b43a could not be found.
  Result2: VM in ERROR state.

  Expected: VM returned to original state, or in shutdown state.
  Expected2: I see no reason to keep dependency from original image for any operation with server. I tested all other operations with server, and they all were succesful, even if base image deleted. Libvirt driver code shows that rescue mode use only 1 property from image_meta to determine bus type for device attached. So I see no reason to raise exception if image_meta data if empty.

  Stack trace for openstack ZED:
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [None req-6611c9f1-63f1-497a-a1bc-51e770d2df09 a27cbf048299483592d84f13e9fe9797 b7df78883aad4ef7ba0a2778c7741462 - - 8a096fed8060480f8f906f57fca19780 8a096fed8060480f8f906f57fca19780] [instance: 73706a9a-e976-4024-8068-c439404ec953] Error trying to Rescue Instance: nova.exception.ImageNotFound: Image be0e302d-79bd-44e1-93a5-68661e93b43a could not be found.
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] Traceback (most recent call last):
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953]   File "/usr/lib/python3/dist-packages/nova/image/glance.py", line 285, in show
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953]     image = self._client.call(context, 2, 'get', args=(image_id,))
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953]   File "/usr/lib/python3/dist-packages/nova/image/glance.py", line 191, in call
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953]     result = getattr(controller, method)(*args, **kwargs)
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953]   File "/usr/lib/python3/dist-packages/glanceclient/v2/images.py", line 197, in get
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953]     return self._get(image_id)
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953]   File "/usr/lib/python3/dist-packages/glanceclient/common/utils.py", line 670, in inner
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953]     return RequestIdProxy(wrapped(*args, **kwargs))
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953]   File "/usr/lib/python3/dist-packages/glanceclient/v2/images.py", line 190, in _get
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953]     resp, body = self.http_client.get(url, headers=header)
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953]   File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 395, in get
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953]     return self.request(url, 'GET', **kwargs)
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953]   File "/usr/lib/python3/dist-packages/glanceclient/common/http.py", line 380, in request
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953]     return self._handle_response(resp)
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953]   File "/usr/lib/python3/dist-packages/glanceclient/common/http.py", line 120, in _handle_response
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953]     raise exc.from_response(resp, resp.content)
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] glanceclient.exc.HTTPNotFound: HTTP 404 Not Found: No image found with ID be0e302d-79bd-44e1-93a5-68661e93b43a
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953]
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] During handling of the above exception, another exception occurred:
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953]
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] Traceback (most recent call last):
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953]   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 4485, in rescue_instance
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953]     self.driver.rescue(context, instance, network_info,
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953]   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 4249, in rescue
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953]     image_meta = objects.ImageMeta.from_image_ref(
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953]   File "/usr/lib/python3/dist-packages/nova/objects/image_meta.py", line 151, in from_image_ref
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953]     image_meta = image_api.get(context, image_ref)
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953]   File "/usr/lib/python3/dist-packages/nova/image/glance.py", line 1205, in get
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953]     return session.show(context, image_id,
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953]   File "/usr/lib/python3/dist-packages/nova/image/glance.py", line 287, in show
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953]     _reraise_translated_image_exception(image_id)

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2002606/+subscriptions