← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1871323] [NEW] cloud-init fails to add default route during _bringup_static_routes

 

Public bug reported:

Cloud Provider: OpenStack (Stein)
Distro: Ubuntu 16.04
Cloud-init version: 19.4-33-gbb4131a2-0ubuntu1~16.04.1

Problem:

Since cloud-init introduced support of classless static route,
cloud-init fails to add route to the gateway in our environment.

Looking through the code, I believe the following code should be patched
as follows.

https://github.com/canonical/cloud-init/blob/master/cloudinit/net/__init__.py#L1113
-            if gateway != "0.0.0.0/0":
+            if gateway != "0.0.0.0":

Can someone verify the issue and give comment on suggested fix?


Here's a sample log of cloud-init with DEBUG flag set.

...
2020-04-07 02:51:55,949 - util.py[DEBUG]: Running command ['ip', '-family', 'inet', 'addr', 'add', '10.54.62.43/32', 'broadcast', '10.54.62.127', 'dev', 'eth0'] with allowed return codes [0] (shell=False, capture=True)
2020-04-07 02:51:55,951 - util.py[DEBUG]: Running command ['ip', '-family', 'inet', 'link', 'set', 'dev', 'eth0', 'up'] with allowed return codes [0] (shell=False, capture=True)
2020-04-07 02:51:55,954 - util.py[DEBUG]: Running command ['ip', '-4', 'route', 'add', '10.54.62.1/32', 'via', '0.0.0.0', 'dev', 'eth0'] with allowed return codes [0] (shell=False, capture=True)
2020-04-07 02:51:55,956 - util.py[DEBUG]: Running command ['ip', '-4', 'route', 'add', '169.254.169.254/32', 'via', '10.54.62.1', 'dev', 'eth0'] with allowed return codes [0] (shell=False, capture=True)
2020-04-07 02:51:55,959 - handlers.py[DEBUG]: finish: init-local/search-OpenStackLocal: FAIL: no local data found from DataSourceOpenStackLocal
2020-04-07 02:51:55,959 - util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceOpenStack.DataSourceOpenStackLocal'> failed
2020-04-07 02:51:55,959 - util.py[DEBUG]: Getting data from <class 'cloudinit.sources.DataSourceOpenStack.DataSourceOpenStackLocal'> failed
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line 760, in find_source
    if s.update_metadata([EventType.BOOT_NEW_INSTANCE]):
  File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line 649, in update_metadata
    result = self.get_data()
  File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line 273, in get_data
    return_value = self._get_data()
  File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceOpenStack.py", line 130, in _get_data
    with EphemeralDHCPv4(self.fallback_interface):
  File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line 57, in __enter__
    return self.obtain_lease()
  File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line 109, in obtain_lease
    ephipv4.__enter__()
  File "/usr/lib/python3/dist-packages/cloudinit/net/__init__.py", line 986, in __enter__
    self._bringup_static_routes()
  File "/usr/lib/python3/dist-packages/cloudinit/net/__init__.py", line 1040, in _bringup_static_routes
    ['dev', self.interface], capture=True)
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2102, in subp
    cmd=args)
cloudinit.util.ProcessExecutionError: Unexpected error while running command.
Command: ['ip', '-4', 'route', 'add', '169.254.169.254/32', 'via', '10.54.62.1', 'dev', 'eth0']
Exit code: 2
Reason: -
Stdout:
Stderr: RTNETLINK answers: Network is unreachable
...

Sample lease file and interface address setup are as follows.

# cat /var/lib/dhcp/eth0.lease
lease {
  interface "eth0";
  fixed-address 10.54.62.43;
  option subnet-mask 255.255.255.255;
  option routers 10.54.62.1;
  option dhcp-lease-time 4294967295;
  option dhcp-message-type 5;
  option domain-name-servers 10.20.30.40;
  option dhcp-server-identifier 10.54.62.1;
  option interface-mtu 1500;
  option rfc3442-classless-static-routes 32,10,54,62,1,0,0,0,0,32,169,254,169,254,10,54,62,1,0,10,54,62,1;
  option broadcast-address 10.54.62.127;
  option host-name "host-10-54-62-43";
  option domain-name "local";
  renew 0 2088/04/25 06:42:22;
  rebind 0 2139/05/10 15:07:51;
  expire 5 2156/05/14 09:56:29;
}

# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr ab:cd:ef:a1:50:a8  
          inet addr:10.54.62.43  Bcast:10.54.62.127  Mask:255.255.255.255
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:12748 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12123 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:50000 
          RX bytes:1757625 (1.7 MB)  TX bytes:1262391 (1.2 MB)

# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.54.62.1      0.0.0.0         UG    0      0        0 eth0
10.54.62.1      0.0.0.0         255.255.255.255 UH    0      0        0 eth0
169.254.169.254 10.54.62.1      255.255.255.255 UGH   0      0        0 eth0

** Affects: cloud-init
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1871323

Title:
  cloud-init fails to add default route during _bringup_static_routes

Status in cloud-init:
  New

Bug description:
  Cloud Provider: OpenStack (Stein)
  Distro: Ubuntu 16.04
  Cloud-init version: 19.4-33-gbb4131a2-0ubuntu1~16.04.1

  Problem:

  Since cloud-init introduced support of classless static route,
  cloud-init fails to add route to the gateway in our environment.

  Looking through the code, I believe the following code should be
  patched as follows.

  https://github.com/canonical/cloud-init/blob/master/cloudinit/net/__init__.py#L1113
  -            if gateway != "0.0.0.0/0":
  +            if gateway != "0.0.0.0":

  Can someone verify the issue and give comment on suggested fix?

  
  Here's a sample log of cloud-init with DEBUG flag set.

  ...
  2020-04-07 02:51:55,949 - util.py[DEBUG]: Running command ['ip', '-family', 'inet', 'addr', 'add', '10.54.62.43/32', 'broadcast', '10.54.62.127', 'dev', 'eth0'] with allowed return codes [0] (shell=False, capture=True)
  2020-04-07 02:51:55,951 - util.py[DEBUG]: Running command ['ip', '-family', 'inet', 'link', 'set', 'dev', 'eth0', 'up'] with allowed return codes [0] (shell=False, capture=True)
  2020-04-07 02:51:55,954 - util.py[DEBUG]: Running command ['ip', '-4', 'route', 'add', '10.54.62.1/32', 'via', '0.0.0.0', 'dev', 'eth0'] with allowed return codes [0] (shell=False, capture=True)
  2020-04-07 02:51:55,956 - util.py[DEBUG]: Running command ['ip', '-4', 'route', 'add', '169.254.169.254/32', 'via', '10.54.62.1', 'dev', 'eth0'] with allowed return codes [0] (shell=False, capture=True)
  2020-04-07 02:51:55,959 - handlers.py[DEBUG]: finish: init-local/search-OpenStackLocal: FAIL: no local data found from DataSourceOpenStackLocal
  2020-04-07 02:51:55,959 - util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceOpenStack.DataSourceOpenStackLocal'> failed
  2020-04-07 02:51:55,959 - util.py[DEBUG]: Getting data from <class 'cloudinit.sources.DataSourceOpenStack.DataSourceOpenStackLocal'> failed
  Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line 760, in find_source
      if s.update_metadata([EventType.BOOT_NEW_INSTANCE]):
    File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line 649, in update_metadata
      result = self.get_data()
    File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line 273, in get_data
      return_value = self._get_data()
    File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceOpenStack.py", line 130, in _get_data
      with EphemeralDHCPv4(self.fallback_interface):
    File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line 57, in __enter__
      return self.obtain_lease()
    File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line 109, in obtain_lease
      ephipv4.__enter__()
    File "/usr/lib/python3/dist-packages/cloudinit/net/__init__.py", line 986, in __enter__
      self._bringup_static_routes()
    File "/usr/lib/python3/dist-packages/cloudinit/net/__init__.py", line 1040, in _bringup_static_routes
      ['dev', self.interface], capture=True)
    File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2102, in subp
      cmd=args)
  cloudinit.util.ProcessExecutionError: Unexpected error while running command.
  Command: ['ip', '-4', 'route', 'add', '169.254.169.254/32', 'via', '10.54.62.1', 'dev', 'eth0']
  Exit code: 2
  Reason: -
  Stdout:
  Stderr: RTNETLINK answers: Network is unreachable
  ...

  Sample lease file and interface address setup are as follows.

  # cat /var/lib/dhcp/eth0.lease
  lease {
    interface "eth0";
    fixed-address 10.54.62.43;
    option subnet-mask 255.255.255.255;
    option routers 10.54.62.1;
    option dhcp-lease-time 4294967295;
    option dhcp-message-type 5;
    option domain-name-servers 10.20.30.40;
    option dhcp-server-identifier 10.54.62.1;
    option interface-mtu 1500;
    option rfc3442-classless-static-routes 32,10,54,62,1,0,0,0,0,32,169,254,169,254,10,54,62,1,0,10,54,62,1;
    option broadcast-address 10.54.62.127;
    option host-name "host-10-54-62-43";
    option domain-name "local";
    renew 0 2088/04/25 06:42:22;
    rebind 0 2139/05/10 15:07:51;
    expire 5 2156/05/14 09:56:29;
  }

  # ifconfig eth0
  eth0      Link encap:Ethernet  HWaddr ab:cd:ef:a1:50:a8  
            inet addr:10.54.62.43  Bcast:10.54.62.127  Mask:255.255.255.255
            UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
            RX packets:12748 errors:0 dropped:0 overruns:0 frame:0
            TX packets:12123 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:50000 
            RX bytes:1757625 (1.7 MB)  TX bytes:1262391 (1.2 MB)

  # route -n
  Kernel IP routing table
  Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
  0.0.0.0         10.54.62.1      0.0.0.0         UG    0      0        0 eth0
  10.54.62.1      0.0.0.0         255.255.255.255 UH    0      0        0 eth0
  169.254.169.254 10.54.62.1      255.255.255.255 UGH   0      0        0 eth0

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1871323/+subscriptions


Follow ups