yahoo-eng-team team mailing list archive

[Chad Smith <1865882@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

On Focal EC2 cloud, Cloud-init v.20.1-5-g67c8e53c-0ubuntu1 recently
redacted header values from logs for security, but the redact operation
was operating on the source headers dictionary. Ec2 API token requests
are mangled with a 'REDACTED' value as a result and cloud-init can no
longer obtain a valid IMDSv2 token and falls back to IMDSv2 after 2
minutes of failed retries.


Commit introducing the regression: 

https://github.com/canonical/cloud-
init/commit/87cd040ed8fe7195cbb357ed3bbf53cd2a81436c#diff-
a779470bb47168497ada0a33f7990b01R284


Error logs of interest
2020-03-03 13:27:47,839 - url_helper.py[DEBUG]: Read from http://169.254.169.254/latest/api/token (400, 341b) after 1 attempts
2020-03-03 13:27:47,839 - DataSourceEc2.py[WARNING]: Calling 'http://169.254.169.254/latest/api/token' failed [0/1s]: bad status code [400]
2020-03-03 13:27:47,847 - url_helper.py[DEBUG]: Please wait 1 seconds while we wait to try again
2020-03-03 13:27:48,848 - DataSourceEc2.py[DEBUG]: Refreshing Ec2 metadata API token
2020-03-03 13:27:48,848 - url_helper.py[DEBUG]: [0/1] open 'http://None/latest/api/token' with {'url': 'http://None/latest/api/token', 'allow_redirects': True, 'method': 'PUT', 'headers': {'User-Agent': 'Cloud-Init/20.1-5-g67c8e53c-0ubuntu1', 'X-aws-ec2-metadata-token-ttl-seconds': 'REDACTED'}} configuration
2020-03-03 13:27:48,849 - DataSourceEc2.py[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc18d46cb10>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
2020-03-03 13:27:48,850 - url_helper.py[DEBUG]: [0/1] open 'http://169.254.169.254/2009-04-04/meta-data/instance-id' with {'url': 'http://169.254.169.254/2009-04-04/meta-data/instance-id', 'allow_redirects': True, 'method': 'GET', 'timeout': 50.0, 'headers': {'User-Agent': 'Cloud-Init/20.1-5-g67c8e53c-0ubuntu1'}} configuration
2020-03-03 13:27:48,869 - url_helper.py[DEBUG]: Read from http://169.254.169.254/2009-04-04/meta-data/instance-id (200, 19b) after 1 attempts
2020-03-03 13:27:48,869 - DataSourceEc2.py[DEBUG]: Using metadata source: 'http://169.254.169.254'
2020-03-03 13:27:48,869 - DataSourceEc2.py[DEBUG]: Refreshing Ec2 metadata API token

** Affects: cloud-init
     Importance: High
     Assignee: Chad Smith (chad.smith)
         Status: Triaged

** Changed in: cloud-init
   Importance: Undecided => High

** Changed in: cloud-init
     Assignee: (unassigned) => Chad Smith (chad.smith)

** Changed in: cloud-init
       Status: New => Triaged

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1865882

Title:
  EC2: token requests 400 due to redacted headers

Status in cloud-init:
  Triaged

Bug description:
  On Focal EC2 cloud, Cloud-init v.20.1-5-g67c8e53c-0ubuntu1 recently
  redacted header values from logs for security, but the redact
  operation was operating on the source headers dictionary. Ec2 API
  token requests are mangled with a 'REDACTED' value as a result and
  cloud-init can no longer obtain a valid IMDSv2 token and falls back to
  IMDSv2 after 2 minutes of failed retries.

  
  Commit introducing the regression: 

  https://github.com/canonical/cloud-
  init/commit/87cd040ed8fe7195cbb357ed3bbf53cd2a81436c#diff-
  a779470bb47168497ada0a33f7990b01R284


  Error logs of interest
  2020-03-03 13:27:47,839 - url_helper.py[DEBUG]: Read from http://169.254.169.254/latest/api/token (400, 341b) after 1 attempts
  2020-03-03 13:27:47,839 - DataSourceEc2.py[WARNING]: Calling 'http://169.254.169.254/latest/api/token' failed [0/1s]: bad status code [400]
  2020-03-03 13:27:47,847 - url_helper.py[DEBUG]: Please wait 1 seconds while we wait to try again
  2020-03-03 13:27:48,848 - DataSourceEc2.py[DEBUG]: Refreshing Ec2 metadata API token
  2020-03-03 13:27:48,848 - url_helper.py[DEBUG]: [0/1] open 'http://None/latest/api/token' with {'url': 'http://None/latest/api/token', 'allow_redirects': True, 'method': 'PUT', 'headers': {'User-Agent': 'Cloud-Init/20.1-5-g67c8e53c-0ubuntu1', 'X-aws-ec2-metadata-token-ttl-seconds': 'REDACTED'}} configuration
  2020-03-03 13:27:48,849 - DataSourceEc2.py[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc18d46cb10>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
  2020-03-03 13:27:48,850 - url_helper.py[DEBUG]: [0/1] open 'http://169.254.169.254/2009-04-04/meta-data/instance-id' with {'url': 'http://169.254.169.254/2009-04-04/meta-data/instance-id', 'allow_redirects': True, 'method': 'GET', 'timeout': 50.0, 'headers': {'User-Agent': 'Cloud-Init/20.1-5-g67c8e53c-0ubuntu1'}} configuration
  2020-03-03 13:27:48,869 - url_helper.py[DEBUG]: Read from http://169.254.169.254/2009-04-04/meta-data/instance-id (200, 19b) after 1 attempts
  2020-03-03 13:27:48,869 - DataSourceEc2.py[DEBUG]: Using metadata source: 'http://169.254.169.254'
  2020-03-03 13:27:48,869 - DataSourceEc2.py[DEBUG]: Refreshing Ec2 metadata API token

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1865882/+subscriptions