yahoo-eng-team team mailing list archive

[Daniel Berrange <1460079@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

When an instance is first created, a copy of the metadata from the
image/volume is saved into the instance system metadata. This provides
an accurate point in time record of the metadata used to configure &
operate instance, even if the metadata on the image is later changed.

For a long while though, much of the code would not use this instance
system metadata, instead just fetching metadata from the image each
time. This had an obvious problem in that if the image was deleted,
those operations would not be able to get image metadata, even though it
was recorded for posterity in the system metadata.

So 2 commits were made to update Nova to fetch system metadata

commit 8e575be75c80ea71a6ad8fb73e6ace1ed708938f
Author: Xavier Queralt <xqueralt@xxxxxxxxxx>
Date: Mon Aug 26 22:53:03 2013 +0200

    Add methods to get image metadata from instance

    This patch adds a couple of utility functions that enclose all the logic
    for getting and parsing the image metadata stored in the instance's
    system metadata.

    First, this will try to fetch the metadata from the real image and will
    prevent it from failing if it is not available. It will be then merged
    with the image metadata stored during the instance creation.

    Related to bug #1039662

    Change-Id: I2130caf19858585571b1199e27f0a98ad5f08701

commit 4389f2292a0177c8eedc0a398ceb3c5535a9ef82
Author: Xavier Queralt <xqueralt@xxxxxxxxxx>
Date: Mon Aug 26 22:55:46 2013 +0200

    Avoid errors on some actions when image not usable

    Using the metadata saved on instance creation, we can now get all the
    image related metadata we need from the instance itself.

    This patch replace the logic for getting the image metadata on some
    actions that shouldn't fail when the image is not accessible (create
    an snapshot, resize, migrate, rescue an instance or attach an
    interface).

    Fixes bug 1039662


Unfortunately the way the compute utils get_image_metadata method was designed, it first fetches the instance system metadata and then fetches the current metadata from the image (if it still exists). The system metadata fields are overwritten by those from the image.

So, there remains a problem that many operations are going to be
performing actions based on the metadata currently associated with the
image, and not that associated with the instance.

By good luck, this does not currently have too many serious ill effects,
but with ever increasing use of image metadata for tuning instance
hardware configuration this is becoming a more pressing problem.

For example, if the hw_disk_bus=virtio when the instance was first
booted, and then the image was later changed to use hw_disk_bus=scsi,
then logic which hotplugs disks may mistakenly end up attempting to
hotplug a SCSI disk instead of a virtio disk which the instance was
initially booted with.

The only code which should look at the image properties should be the
initial boot operation. There after all code should be using the
recorded instance system metadata, so it is making decisions that are
consistent with those made when the instance was first booted.

There is an exception for the rescue operation, which by its very nature
should be using the metadata from the rescue image, not the original
instance system meta, since the hardware configuration needs to match
that of the rescue image requirements.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1460079

Title:
  Instance system metadata is sometimes overwritten by image metadata

Status in OpenStack Compute (Nova):
  New

Bug description:
  When an instance is first created, a copy of the metadata from the
  image/volume is saved into the instance system metadata. This provides
  an accurate point in time record of the metadata used to configure &
  operate instance, even if the metadata on the image is later changed.

  For a long while though, much of the code would not use this instance
  system metadata, instead just fetching metadata from the image each
  time. This had an obvious problem in that if the image was deleted,
  those operations would not be able to get image metadata, even though
  it was recorded for posterity in the system metadata.

  So 2 commits were made to update Nova to fetch system metadata

  commit 8e575be75c80ea71a6ad8fb73e6ace1ed708938f
  Author: Xavier Queralt <xqueralt@xxxxxxxxxx>
  Date: Mon Aug 26 22:53:03 2013 +0200

      Add methods to get image metadata from instance

      This patch adds a couple of utility functions that enclose all the logic
      for getting and parsing the image metadata stored in the instance's
      system metadata.

      First, this will try to fetch the metadata from the real image and will
      prevent it from failing if it is not available. It will be then merged
      with the image metadata stored during the instance creation.

      Related to bug #1039662

      Change-Id: I2130caf19858585571b1199e27f0a98ad5f08701

  commit 4389f2292a0177c8eedc0a398ceb3c5535a9ef82
  Author: Xavier Queralt <xqueralt@xxxxxxxxxx>
  Date: Mon Aug 26 22:55:46 2013 +0200

      Avoid errors on some actions when image not usable

      Using the metadata saved on instance creation, we can now get all the
      image related metadata we need from the instance itself.

      This patch replace the logic for getting the image metadata on some
      actions that shouldn't fail when the image is not accessible (create
      an snapshot, resize, migrate, rescue an instance or attach an
      interface).

      Fixes bug 1039662

  
  Unfortunately the way the compute utils get_image_metadata method was designed, it first fetches the instance system metadata and then fetches the current metadata from the image (if it still exists). The system metadata fields are overwritten by those from the image.

  So, there remains a problem that many operations are going to be
  performing actions based on the metadata currently associated with the
  image, and not that associated with the instance.

  By good luck, this does not currently have too many serious ill
  effects, but with ever increasing use of image metadata for tuning
  instance hardware configuration this is becoming a more pressing
  problem.

  For example, if the hw_disk_bus=virtio when the instance was first
  booted, and then the image was later changed to use hw_disk_bus=scsi,
  then logic which hotplugs disks may mistakenly end up attempting to
  hotplug a SCSI disk instead of a virtio disk which the instance was
  initially booted with.

  The only code which should look at the image properties should be the
  initial boot operation. There after all code should be using the
  recorded instance system metadata, so it is making decisions that are
  consistent with those made when the instance was first booted.

  There is an exception for the rescue operation, which by its very
  nature should be using the metadata from the rescue image, not the
  original instance system meta, since the hardware configuration needs
  to match that of the rescue image requirements.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1460079/+subscriptions