yahoo-eng-team team mailing list archive

[OpenStack Infra <1989514@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/cinder/+/857528
Committed: https://opendev.org/openstack/cinder/commit/56abc9d5b9a07ca31791e385d93ca5b56b6bd74e
Submitter: "Zuul (22348)"
Branch:    master

commit 56abc9d5b9a07ca31791e385d93ca5b56b6bd74e
Author: melanie witt <melwittt@xxxxxxxxx>
Date:   Thu Sep 15 00:00:31 2022 +0000

    NFS update volume attachment format during volume snapshot
    
    During a NFS volume snapshot of an attached volume, a QCOW2 snapshot is
    created and is made the active volume for the instance. The associated
    volume attachment is however not updated, resulting in an unbootable
    instance after a stop/start.
    
    This adds an update to the volume format and attachment connection_info
    format during the snapshot.
    
    Closes-Bug: #1989514
    
    Change-Id: I3c638dab2f0b5a2128eaeeea533450d7cdfa1577


** Changed in: cinder
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1989514

Title:
  NFS volume snapshot does not update volume attachment format to qcow2

Status in Cinder:
  Fix Released
Status in OpenStack Compute (nova):
  Invalid

Bug description:
  This was reported downstream when a customer hit the issue [1].

  They found that after performing a volume snapshot on a stopped
  instance, that instance could no longer boot.

  The instance XML had the correct disk format "qcow2" after the
  snapshot while it was still stopped. However, when someone then tries
  to start the instance, the regenerated XML contains the wrong disk
  format "raw" (nova regenerates fresh XML on hard reboot, boot from
  shutoff) and that prevents the instance from booting.

  While tracing the issue I noticed that after the snapshot, the volume
  attachment still showed the format "raw". It should have been "qcow2"
  after the snapshot, I think.

  The customer found that rebuilding the instance resulted in a booting
  instance despite the underlying volume (snapshot) being the same.
  Rebuild generates a new volume attachment and that volume attachment
  reflected the correct format "qcow2".

  Based on this, it appeared to me that volume attachment format was not
  being updated from "raw" to "qcow2" after the volume snapshot was
  created as qcow2 and was made the active volume  for the instance.

  AFAICT, there is not a way for nova to update the volume attachment
  format.

  Repro steps:

    $ openstack volume create --image
  f8e5b23e-b1d3-44e8-bc71-ac7aca4f6f8e --size 1 --bootable my_volume

    $ openstack server create --flavor c1  --volume my_volume --network
  private --wait my_server

    $ openstack server stop d97a6178-112d-487b-ba21-1af2a4a78456

    (Instance has disk format "raw" and volume attachment also shows format "raw")
    (To check this:
    
      $ virsh dumpxml d97a6178-112d-487b-ba21-1af2a4a78456

      $ openstack --os-volume-api-version 3.27 volume attachment show 2f43581f-ba30-423e-ba5f-7b0ac7768fa5 -f json
    )

    $ openstack volume snapshot create --volume d5c71e24-46ac-49be-
  bf33-667598b6058b --force my_volume_snapshot

    (Instance has correct disk format "qcow2" at this point, volume
  attachment shows format "raw")

    $ openstack server start d97a6178-112d-487b-ba21-1af2a4a78456

    (Instance now has incorrect disk format "raw" and volume attachment
  still shows format "raw")

  To see the effect of a rebuild:

    $ openstack server rebuild d97a6178-112d-487b-ba21-1af2a4a78456
  --image f8e5b23e-b1d3-44e8-bc71-ac7aca4f6f8e --wait

    (Instance has disk format "qcow2" and volume attachment also shows format "qcow2")
    (Volume attachment is newly created with new ID etc)

  [1] https://bugzilla.redhat.com/show_bug.cgi?id=2118150

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1989514/+subscriptions