yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1870228] [NEW] cloud-init metadata fallback broken

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: James Denton <james.denton@xxxxxxxxxxx>
Date: Wed, 01 Apr 2020 21:52:06 -0000
Reply-to: Bug 1870228 <1870228@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I came across an issue today for a user that was experiencing issues
connecting to metadata at 169.254.169.254. For a long time, cloud-init
has had a fallback mechanism to that allowed it to contact the metadata
service at http://<dhcp server ip>/latest/meta-data if
http://169.254.169.254/latest/meta-data were unavailable, like so:

[  157.574921] cloud-init[1313]: 2020-03-31 09:53:24,158 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [50/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x7fa9e07c5890>, 'Connection to 169.254.169.254 timed out. (connect timeout=50.0)'))]
[  208.629083] cloud-init[1313]: 2020-03-31 09:54:15,214 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [101/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x7fa9e07c5350>, 'Connection to 169.254.169.254 timed out. (connect timeout=50.0)'))]
[  226.639267] cloud-init[1313]: 2020-03-31 09:54:33,224 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [119/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x7fa9e07c5250>, 'Connection to 169.254.169.254 timed out. (connect timeout=17.0)'))]
[  227.640812] cloud-init[1313]: 2020-03-31 09:54:34,225 - DataSourceEc2.py[CRITICAL]: Giving up on md from ['http://169.254.169.254/2009-04-04/meta-data/instance-id'] after 120 seconds
[  227.651134] cloud-init[1313]: 2020-03-31 09:54:34,236 - url_helper.py[WARNING]: Calling 'http://10.19.48.2/latest/meta-data/instance-id' failed [0/120s]: request error [('Connection aborted.', error(111, 'Connection refused'))]
[  228.655226] cloud-init[1313]: 2020-03-31 09:54:35,240 - url_helper.py[WARNING]: Calling 'http://10.19.48.2/latest/meta-data/instance-id' failed [1/120s]: request error [('Connection aborted.', error(111, 'Connection refused'))]

In this Stein environment, isolated metadata is enabled, and the qdhcp
namespace has a listener at 169.254.169.254:80. Previous versions of
Neutron had the listener on 0.0.0.0:80, which helped facilitate the
fallback mechanism described above. The bug/patch where this was changed
is here:

[1] https://bugs.launchpad.net/neutron/+bug/1745618

Having this functionality back would be nice. Thoughts?

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1870228

Title:
  cloud-init metadata fallback broken

Status in neutron:
  New

Bug description:
  I came across an issue today for a user that was experiencing issues
  connecting to metadata at 169.254.169.254. For a long time, cloud-init
  has had a fallback mechanism to that allowed it to contact the
  metadata service at http://<dhcp server ip>/latest/meta-data if
  http://169.254.169.254/latest/meta-data were unavailable, like so:

  [  157.574921] cloud-init[1313]: 2020-03-31 09:53:24,158 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [50/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x7fa9e07c5890>, 'Connection to 169.254.169.254 timed out. (connect timeout=50.0)'))]
  [  208.629083] cloud-init[1313]: 2020-03-31 09:54:15,214 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [101/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x7fa9e07c5350>, 'Connection to 169.254.169.254 timed out. (connect timeout=50.0)'))]
  [  226.639267] cloud-init[1313]: 2020-03-31 09:54:33,224 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [119/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x7fa9e07c5250>, 'Connection to 169.254.169.254 timed out. (connect timeout=17.0)'))]
  [  227.640812] cloud-init[1313]: 2020-03-31 09:54:34,225 - DataSourceEc2.py[CRITICAL]: Giving up on md from ['http://169.254.169.254/2009-04-04/meta-data/instance-id'] after 120 seconds
  [  227.651134] cloud-init[1313]: 2020-03-31 09:54:34,236 - url_helper.py[WARNING]: Calling 'http://10.19.48.2/latest/meta-data/instance-id' failed [0/120s]: request error [('Connection aborted.', error(111, 'Connection refused'))]
  [  228.655226] cloud-init[1313]: 2020-03-31 09:54:35,240 - url_helper.py[WARNING]: Calling 'http://10.19.48.2/latest/meta-data/instance-id' failed [1/120s]: request error [('Connection aborted.', error(111, 'Connection refused'))]

  In this Stein environment, isolated metadata is enabled, and the qdhcp
  namespace has a listener at 169.254.169.254:80. Previous versions of
  Neutron had the listener on 0.0.0.0:80, which helped facilitate the
  fallback mechanism described above. The bug/patch where this was
  changed is here:

  [1] https://bugs.launchpad.net/neutron/+bug/1745618

  Having this functionality back would be nice. Thoughts?

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1870228/+subscriptions