kernel-packages team mailing list archive

Thread
Date

[Bug 1551419] Re: Fix UUID endianness patch breaks cloud-init on Azure

To: kernel-packages@xxxxxxxxxxxxxxxxxxx
From: Dan Watkins <daniel.watkins@xxxxxxxxxxxxx>
Date: Wed, 02 Mar 2016 17:30:26 -0000
Reply-to: Bug 1551419 <1551419@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx

** Description changed:

On Azure, cloud-init relies on the system-uuid as based by SMBIOS a
unique ID for a cloud instance. If this ID ever changes, then cloud-
init will attempt to reprovision the VM.

- This recent kernel patch in the Ubuntu kernel corrects the endianness
- for some SMBIOS fields, but also has the effect causing cloud-init to
- think that the system-uuid has changed:
+ This recent kernel patch in the Ubuntu kernel incorrectly modifies the
+ endianness for some SMBIOS fields, which has the effect causing cloud-
+ init to think that the system-uuid has changed:
http://kernel.ubuntu.com/git/ubuntu/ubuntu-
trusty.git/commit/drivers/firmware?id=3ec24c55be6c543797ba3ee9a227a5631aef607e

- The impact is that cloud-init attempts to reprovision the VM, often
- causing the customer to lose access to their VM.
+ cloud-init needs to consider both the reported UUID and the "first three
+ fields endian-reversed" UUID as the same, so that users shifting between
+ unaffected kernels and affected kernels, or affected kernels and fixed
+ kernels do not see their instances reprovisioned.
+
+ [Impact]
+
+ The impact is that cloud-init attempts to reprovision VMs when they
+ reboot to use the new kernel, often causing the customer to lose access
+ to their VM.
+
+ Once the kernel is fixed, rebooting from an affected kernel to the new
+ kernel will have the same effect.
+
+ [Test Case]
+
+ Failure:
+
+ 1) Boot an Azure instance using an image with a pre-broken kernel (e.g. b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04_3-LTS-amd64-server-20160201-en-us-30GB)
+ 2) Upgrade the kernel and reboot.
+ 3) SSH to the instance; you will observe that you are prompted to change SSH host keys because cloud-init has run again.
+
+ Success:
+
+ 1) Boot an Azure instance using an image with a pre-broken kernel (e.g. b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04_3-LTS-amd64-server-20160201-en-us-30GB)
+ 2) Install the new version of cloud-init.
+ 3) Upgrade the kernel and reboot.
+ 4) Observe that you are not prompted when SSHing to instance, as cloud-init has not run again.
+
+ [Regression Potential]
+
+ The change is limited to the Azure data source. It affects how instance
+ IDs are determined, but the change does so in a limited way.

** Description changed:

On Azure, cloud-init relies on the system-uuid as based by SMBIOS a
unique ID for a cloud instance. If this ID ever changes, then cloud-
init will attempt to reprovision the VM.

This recent kernel patch in the Ubuntu kernel incorrectly modifies the
endianness for some SMBIOS fields, which has the effect causing cloud-
init to think that the system-uuid has changed:
http://kernel.ubuntu.com/git/ubuntu/ubuntu-
trusty.git/commit/drivers/firmware?id=3ec24c55be6c543797ba3ee9a227a5631aef607e

cloud-init needs to consider both the reported UUID and the "first three
fields endian-reversed" UUID as the same, so that users shifting between
unaffected kernels and affected kernels, or affected kernels and fixed
kernels do not see their instances reprovisioned.

[Impact]

The impact is that cloud-init attempts to reprovision VMs when they
reboot to use the new kernel, often causing the customer to lose access
to their VM.

Once the kernel is fixed, rebooting from an affected kernel to the new
kernel will have the same effect.

[Test Case]

Failure:

1) Boot an Azure instance using an image with a pre-broken kernel (e.g. b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04_3-LTS-amd64-server-20160201-en-us-30GB)
2) Upgrade the kernel and reboot.
3) SSH to the instance; you will observe that you are prompted to change SSH host keys because cloud-init has run again.

- Success:
+ Success (upgrade from not broken->broken):

1) Boot an Azure instance using an image with a pre-broken kernel (e.g. b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04_3-LTS-amd64-server-20160201-en-us-30GB)
2) Install the new version of cloud-init.
3) Upgrade the kernel and reboot.
4) Observe that you are not prompted when SSHing to instance, as cloud-init has not run again.

+ Success (upgrade from broken->fixed):
+ 1) Boot an Azure instance using an image with a broken kernel (e.g. b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04_4-LTS-amd64-server-20160222-en-us-30GB)
+ 2) Install the new version of cloud-init.
+ 3) Upgrade to the fixed kernel (once it is available) and reboot.
+ 4) Observe that you are not prompted when SSHing to instance, as cloud-init has not run again.
+
+ Success (upgrade from not broken->fixed):
+ 1) Boot an Azure instance using an image with a pre-broken kernel (e.g. b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04_3-LTS-amd64-server-20160201-en-us-30GB)
+ 2) Install the new version of cloud-init.
+ 3) Upgrade to the fixed kernel (once it is available) and reboot.
+ 4) Observe that you are not prompted when SSHing to instance, as cloud-init has not run again.
+
[Regression Potential]

The change is limited to the Azure data source. It affects how instance
IDs are determined, but the change does so in a limited way.

--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1551419

Title:
Fix UUID endianness patch breaks cloud-init on Azure

Status in cloud-init package in Ubuntu:
In Progress
Status in linux package in Ubuntu:
Confirmed
Status in cloud-init source package in Trusty:
New
Status in linux source package in Trusty:
Fix Committed
Status in cloud-init source package in Vivid:
New
Status in linux source package in Vivid:
Fix Committed

Bug description:
On Azure, cloud-init relies on the system-uuid as based by SMBIOS a
unique ID for a cloud instance. If this ID ever changes, then cloud-
init will attempt to reprovision the VM.

cloud-init needs to consider both the reported UUID and the "first
three fields endian-reversed" UUID as the same, so that users shifting
between unaffected kernels and affected kernels, or affected kernels
and fixed kernels do not see their instances reprovisioned.

[Impact]

The impact is that cloud-init attempts to reprovision VMs when they
reboot to use the new kernel, often causing the customer to lose
access to their VM.

Once the kernel is fixed, rebooting from an affected kernel to the new
kernel will have the same effect.

[Test Case]

Failure:

Success (upgrade from not broken->broken):

Success (upgrade from broken->fixed):
1) Boot an Azure instance using an image with a broken kernel (e.g. b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04_4-LTS-amd64-server-20160222-en-us-30GB)
2) Install the new version of cloud-init.
3) Upgrade to the fixed kernel (once it is available) and reboot.
4) Observe that you are not prompted when SSHing to instance, as cloud-init has not run again.

Success (upgrade from not broken->fixed):
1) Boot an Azure instance using an image with a pre-broken kernel (e.g. b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04_3-LTS-amd64-server-20160201-en-us-30GB)
2) Install the new version of cloud-init.
3) Upgrade to the fixed kernel (once it is available) and reboot.
4) Observe that you are not prompted when SSHing to instance, as cloud-init has not run again.

[Regression Potential]

The change is limited to the Azure data source. It affects how
instance IDs are determined, but the change does so in a limited way.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1551419/+subscriptions

References

[Bug 1551419] [NEW] Fix UUID endianness patch breaks cloud-init on Azure
From: Stephen A. Zarkos, 2016-02-29