yahoo-eng-team team mailing list archive

[Chad Smith <1800223@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

cloud-init 18.4-7-g4652b196-0ubuntu1 on Azure does not retry IMDS if the
service is not yet available.


Per configuration in the azure datasource, it should retry 4 times before failing, but logs only list one attempt at  http://169.254.169.254/metadata/instance?api-version=2017-12-01


We can see this in the following excerpt of /var/log/cloud-init.log

2018-10-26 17:26:51,162 - url_helper.py[DEBUG]: [0/4] open 'http://169.254.169.254/metadata/instance?api-version=2017-12-01' with {'url': 'http://169.254.169.254/metadata/instance?api-version=2017-12-01', 'allow_redirects': True, 'method': 'GET', 'timeout': 1.0, 'headers': {'User-Agent': 'Cloud-Init/18.4-7-g4652b196-0ubuntu1', 'Metadata': 'true'}} configuration
2018-10-26 17:26:52,168 - DataSourceAzure.py[DEBUG]: Ignoring IMDS instance metadata: HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /metadata/instance?api-version=2017-12-01 (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f392a7c5668>, 'Connection to 169.254.169.254 timed out. (connect timeout=1.0)'))
2018-10-26 17:26:52,168 - util.py[DEBUG]: Crawl of Azure Instance Metadata Service (IMDS) took 1.007 seconds


Only 1 attempt (out of 4) was attempted and then Azure datasource gave up after receiving a ConnectTimeoutError.

At a later time on the instance, I was able to see that the IMDS service
was available using the following python script:

$ python3 -c 'from cloudinit.sources.DataSourceAzure import
get_metadata_from_imds; print(get_metadata_from_imds("eth0",
retries=2))';

{'compute': {'location': 'eastus2', 'name': 'my-c1', 'offer':
'UbuntuServer', 'osType': 'Linux', 'placementGroupId': '',
'platformFaultDomain': '0', 'platformUpdateDomain': '0', 'publisher':
'Canonical', 'resourceGroupName': 'srugroup1', 'sku': '18.10-DAILY',
'subscriptionId': '12aad61c-6de4-4e53-a6c6-5aff52a83777', 'tags': '',
'version': '18.10.201810230', 'vmId': 'ec22ac8c-f728-425f-852f-
5b86d0c9f2f7', 'vmScaleSetName': '', 'vmSize': 'Standard_DS1_v2',
'zone': ''}, 'network': {'interface': [{'ipv4': {'ipAddress':
[{'privateIpAddress': '10.0.0.4', 'publicIpAddress': '40.70.24.192'}],
'subnet': [{'address': '10.0.0.0', 'prefix': '24'}]}, 'ipv6':
{'ipAddress': []}, 'macAddress': '000D3A03582B'}]}}

** Affects: cloud-init
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1800223

Title:
  Azure: IMDS service not retried on timeout

Status in cloud-init:
  New

Bug description:
  cloud-init 18.4-7-g4652b196-0ubuntu1 on Azure does not retry IMDS if
  the service is not yet available.

  
  Per configuration in the azure datasource, it should retry 4 times before failing, but logs only list one attempt at  http://169.254.169.254/metadata/instance?api-version=2017-12-01

  
  We can see this in the following excerpt of /var/log/cloud-init.log

  2018-10-26 17:26:51,162 - url_helper.py[DEBUG]: [0/4] open 'http://169.254.169.254/metadata/instance?api-version=2017-12-01' with {'url': 'http://169.254.169.254/metadata/instance?api-version=2017-12-01', 'allow_redirects': True, 'method': 'GET', 'timeout': 1.0, 'headers': {'User-Agent': 'Cloud-Init/18.4-7-g4652b196-0ubuntu1', 'Metadata': 'true'}} configuration
  2018-10-26 17:26:52,168 - DataSourceAzure.py[DEBUG]: Ignoring IMDS instance metadata: HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /metadata/instance?api-version=2017-12-01 (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f392a7c5668>, 'Connection to 169.254.169.254 timed out. (connect timeout=1.0)'))
  2018-10-26 17:26:52,168 - util.py[DEBUG]: Crawl of Azure Instance Metadata Service (IMDS) took 1.007 seconds

  
  Only 1 attempt (out of 4) was attempted and then Azure datasource gave up after receiving a ConnectTimeoutError.

  At a later time on the instance, I was able to see that the IMDS
  service was available using the following python script:

  $ python3 -c 'from cloudinit.sources.DataSourceAzure import
  get_metadata_from_imds; print(get_metadata_from_imds("eth0",
  retries=2))';

  {'compute': {'location': 'eastus2', 'name': 'my-c1', 'offer':
  'UbuntuServer', 'osType': 'Linux', 'placementGroupId': '',
  'platformFaultDomain': '0', 'platformUpdateDomain': '0', 'publisher':
  'Canonical', 'resourceGroupName': 'srugroup1', 'sku': '18.10-DAILY',
  'subscriptionId': '12aad61c-6de4-4e53-a6c6-5aff52a83777', 'tags': '',
  'version': '18.10.201810230', 'vmId': 'ec22ac8c-f728-425f-852f-
  5b86d0c9f2f7', 'vmScaleSetName': '', 'vmSize': 'Standard_DS1_v2',
  'zone': ''}, 'network': {'interface': [{'ipv4': {'ipAddress':
  [{'privateIpAddress': '10.0.0.4', 'publicIpAddress': '40.70.24.192'}],
  'subnet': [{'address': '10.0.0.0', 'prefix': '24'}]}, 'ipv6':
  {'ipAddress': []}, 'macAddress': '000D3A03582B'}]}}

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1800223/+subscriptions