cloud-init team mailing list archive

Thread
Date

Looking for advice on cloud-init problem starting new Ubuntu 24 Openstack instance

To: "cloud-init@xxxxxxxxxxxxxxxxxxx" <cloud-init@xxxxxxxxxxxxxxxxxxx>
From: "John Pfuntner -X (jpfuntne - ACTALENT SERVICES LLC at Cisco)" <jpfuntne@xxxxxxxxx>
Date: Wed, 05 Mar 2025 14:34:17 -0000
Accept-language: en-US
Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=cisco.com; dmarc=pass action=none header.from=cisco.com; dkim=pass header.d=cisco.com; arc=none
Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=s3fbC4sySh9swmQVwgymSY1StvWHKb9z+1qoos3NhKc=; b=gwf/cK8YOvriTaXPdbNmrxZX5bh50XZjkMbTuvNlilkMb3mOl/Sf+f9EvY+S6shVHuE5qqtnfHjzFiIuYLfs4Ok/tFf/3uewnEjUosnDBF6y9zFS5msEZOVkETWi3cxIU+NRxDBtbYEkL0iDnLaB4xvVg+4Ihr4w/fMKHD9ys5/2v4errZhs6XGn+XrFLMQeVa44hDgrEJv3nsoK1gHA7YVeh90Qo3oqghSCVyX/2/xDpuoiWqmVLxjuSTpjxTHBOzwlsq2tV8+L8nLjiGXOU6BLgtA5omlXDH+NDPdacDrHr7YX52sod6rajt/7mdl9HvvN0kRbnJJEA9QKnvIfXg==
Arc-seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=JjAT5YY+aL2dbEDsgGhRsvlCqYXL7uPK/I2Yzw7owxZZy3sT8rJaN9sQ5MPGSvaezc5T0HMAC2AWQ5V60cC24pq/qG1BvcqoXZAQikVeZTIbtW1IWO8aNaLVdQ0YpVtu3taTILmWBXF104tHmvU0RmCWvrD9DNJbKkK3J5BFHomhvOdEnVkgi47IJLdf4kK0wUMh8eBNL8mhAPQOd4XoZi0c+lD0eDgn2a5ruGpW40LBZPd/hKxu0bupt2/kgvnfbuiFAc1avDOdAG/hgbKUcYr8EOQ33yr6SE9l4YrVUrH3C8De2pMe5K48zvAco+K0oGr+Z9AkCiknH7gkHeINJA==
Authentication-results: alln-opgw-1.cisco.com; dkim=pass (signature verified) header.i=@cisco.com
Ironport-data: A9a23:4pfkVanTUu/UaBplw26HaC/o5gzQJ0RdPkR7XQ2eYbSJt1+Wr1Gzt xJNXG/VM/3fZ2GhLtsla9jnpExSvMfdz4dhSAFkqXs9EVtH+JHPbTi7wugcHM8zwunrFh8PA xA2M4GYRCwMZiaC4E/rav658CEUOZigHtLUEPTDNj16WThqQSIgjQMLs+Mii+aEu/Dha++2k Y20+pG31GONgWYubzpPs//b83uDgdyr0N8mlg1mDRx0lAe2e0k9VPo3Oay3Jn3kdYhYdsbSb /rD1ryw4lTC9B4rDN6/+p6jGqHdauePVeQmoiM+t5mK2nCulARrukoIHKZ0hXNsttm8t4sZJ OOhGnCHYVxB0qXkwIzxWvTDes10FfUuFLTveRBTvSEPpqHLWyOE/hlgMK05FYkK88FQLF4Xz KY3MTEfbki+udmK7b3uH4GAhux7RCXqFJkUtnclyXTSCuwrBMiTBa7L/tRfmjw3g6iiH96HO JFfMmUpNkmdJUQTZz/7C7pm9AusrnvnfSBetFSYjaE2+GPUigd21dABNfKOIIXSGpsJwBjwS mTuxV7TEB9dMdmlzBGH/3KpgMyVgT3lcddHfFG/3rsw6LGJ/UQYDBoLWR66p/ilokq/QM5Eb Ugd5ywlrbA7+AqtQ8WVYvGjiGSPshhZX59bFPc3rVndjKHV+A2eQGMDS1atdeAbiSP/fhRzv nehlNLyDjspu7qQIU9xPJ/Nxd9uEUD59VM/WBI=
Ironport-hdrordr: A9a23:ADcMlq8hJXl0CWT35Kduk+F+db1zdoMgy1knxilNoENuA6+lfp GV/MjziyWUtN9IYgBcpTnhAsW9qADnhOJICOgqTPyftWbdyRCVxe1ZnO/fKl7bamfDH4xmpN 1dmsFFYbWaYykZ/KPHCWKDYrAdKay8gdiVbJDlvg5QpG9RGsZdxjY8IDy2VmdxQw5LD4c4Ep 2z2uprzgDLRV0nKu6AKj0gRebsm/3n/aiKXfcBPXEawTjLqQntxK/xEhCe0BtbeShI260e/W /MlBG8zrm/stmgoyWsm1P73tBzop/M29FDDMuDhow+MTP3kDulY4xnRvmroC01muey81wn+e O87CvIfv4DpU85TFvF4ScF6DOQkwrGLEWSi2NwtEGT4PARgghKTvapy7gpNicxoHBQw+2UmJ g7oV5x8aAnQC8pWE/Glp71v9YArDvynVMy1eEUlHBRSo0YdftYqpEe5lpcFNMaEDv9851PKp gZMCjw3ocjTbqhVQGvgkB/hNi3GngjFBaPRUYP/sSTzjhNhXh8i08V3tYWkHsM/I80D8As3Z WNDo140LVVCsMGZ6N0A+kMBcOxF2zWWBrJdGafO07uGq0LM2/E75T3/LI27ue3f4Fg9up6pL 3RFFdD8WIicUPnDsODmJVN7xDWWW24GS/gz8lPjqIJz4EUhICbQxFrZGpe4PdI+c9vc/Ezc8 zDSK5rPw==
Ironport-phdr: A9a23:du3CLhGH6oe5M7Tso9SK4J1GfhMY04WdBeZdwpMjj7QLdbys4NG7e kfe/v5qylTOWNaT5/FFjr/Ourv7ESwb4JmHuWwfapEESRIfiMsXkgBhSM6IAEH2NrjrOgQxH d9JUxlu+HTTDA==
Thread-index: AduN2JKWNWLxsdDVQoG7+yIdaG5Y1g==
Thread-topic: Looking for advice on cloud-init problem starting new Ubuntu 24 Openstack instance

Lately I'm experiencing some occasional issues connecting via ssh to Ubuntu 24 Openstack instances and could use some ideas about how to correct the behavior so there are no connection issues.  We get images from https://cloud-images.ubuntu.com/noble/ (such as https://cloud-images.ubuntu.com/noble/20250226/noble-server-cloudimg-amd64.img).  The problems happen about 25-33% of the time and seem to be related to getting the metadata to retrieve the public key to put in ~ubuntu/.ssh/authorized_keys.  I first noticed an issue on 2025-02-22 in some tooling of my team. There are a couple of different symptoms I've seen:

Symptoms 1
I believe the log is taken from an instance created with the 20250226 image.
ssh behavior
Permission denied (publickey)
Note: I didn't record a complete error message
Log events from Openstack console
[    4.424302] cloud-init[640]: Cloud-init v. 24.4-0ubuntu1~24.04.2 running 'init-local' at Thu, 27 Feb 2025 18:54:14 +0000. Up 4.41 seconds.
[    6.206522] cloud-init[640]: 2025-02-27 18:54:16,207 - url_helper.py[ERROR]: Timed out, no response from urls: ['http://[fe80::a9fe:a9fe%25ens3]/openstack', 'http://169.254.169.254/openstack']
[    6.211260] cloud-init[640]: 2025-02-27 18:54:16,212 - log_util.py[WARNING]: No active metadata service found
.
.
.
[   12.441227] cloud-init[1120]: Cloud-init v. 24.4-0ubuntu1~24.04.2 running 'modules:final' at Thu, 27 Feb 2025 18:54:22 +0000. Up 12.42 seconds.
ci-info: no authorized SSH keys fingerprints found for user ubuntu.

Symptoms 2
The log is taken from an instance created with the 20250304 image.
ssh behavior
ssh: connect to host pfuntner-ubuntu24-20250304-3-5 port 22: Connection refused
Log events from Openstack console
[    4.863606] cloud-init[600]: Cloud-init v. 24.4.1-0ubuntu0~24.04.1 running 'init-local' at Wed, 05 Mar 2025 12:17:19 +0000. Up 4.85 seconds.
[    6.691475] cloud-init[600]: 2025-03-05 12:17:21,715 - url_helper.py[WARNING]: Ec2 IMDS endpoint returned a 503 error. HTTP endpoint is overloaded. Retrying.
[    7.694753] cloud-init[600]: 2025-03-05 12:17:22,718 - url_helper.py[WARNING]: Ec2 IMDS endpoint returned a 503 error. HTTP endpoint is overloaded. Retrying.

The warning message is repeated every second.  I left one instance up and it's been retrying for over 20 minutes and doesn't get further in the boot.

I opened https://github.com/canonical/cloud-init/issues/6062 with some complete logs but I don't think anyone has looked at it.

Our Openstack admin team basically said we're on our own since we're importing images rather than the images they provide (they don't provide Ubuntu 24 yet).  I don't know if there's a bug in Openstack, cloud-init or something else.  Does anyone have any advice?