yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1982882] [NEW] Failed to process compatible router - network is unreachable

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Adam Gorski <1982882@xxxxxxxxxxxxxxxxxx>
Date: Tue, 26 Jul 2022 19:26:37 -0000
Reply-to: Bug 1982882 <1982882@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

I am using kolla-ansible to install OpenStack on 3 servers:

Controller
Compute1
Compute2

Each server has three network interfaces:
1. management (10.0.0.0/24)
2. provider (xxx.xxx.xxx.160/27)
3. server internet access (xxx.xxx.xxx.160/27)

After deployment, I am unable to get internet connectivity on any
instance. The following error is seen in
/var/log/kolla/neutron/neutron-l3-agent.log after instance creation. I
am unsure what changes I can make to resolve this. Network and router
information is below.


LOG:

2022-07-26 18:29:26.061 18 INFO neutron.agent.l3.agent [-] Starting processing update 4e471427-5bf3-4e72-9e56-cea3c8474b0c, action 3, priority 2, update_id 0f830a94-0c6f-4dff-9063-aabad002173a. Wait time elapsed: 6.087
2022-07-26 18:29:26.061 18 INFO neutron.agent.l3.agent [-] Starting router update for 4e471427-5bf3-4e72-9e56-cea3c8474b0c, action 3, priority 2, update_id 0f830a94-0c6f-4dff-9063-aabad002173a. Wait time elapsed: 6.087
2022-07-26 18:29:26.492 18 INFO neutron.agent.linux.interface [-] Device qg-6a95a75b-9b already exists
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info [-] (101, 'Network is unreachable'): pr2modules.netlink.exceptions.NetlinkError: (101, 'Network is unreachable')
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/common/utils.py", line 174, in call
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info     return func(*args, **kwargs)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 1312, in process
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info     self.process_external()
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info   File "<decorator-gen-92>", line 2, in process_external
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/common/coordination.py", line 78, in _synchronized
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info     return f(*a, **k)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 1046, in process_external
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info     self._process_external_gateway(ex_gw_port)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 927, in _process_external_gateway
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info     self.external_gateway_added(ex_gw_port, interface_name)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 880, in external_gateway_added
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info     ex_gw_port, interface_name, self.ns_name, preserve_ips)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 823, in _external_gateway_added
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info     ns_name, preserve_ips)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 853, in _external_gateway_settings
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info     device.route.add_gateway(ip)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 613, in add_gateway
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info     scope=scope)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 651, in add_route
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info     table=table, metric=metric, scope=scope, **kwargs)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 1525, in add_ip_route
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info     metric=metric, scope=scope, proto=proto, **kwargs)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 272, in _wrap
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info     r_call_timeout)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 215, in remote_call
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info     raise exc_type(*result[2])
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info pr2modules.netlink.exceptions.NetlinkError: (101, 'Network is unreachable')
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info
2022-07-26 18:29:26.977 18 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 4e471427-5bf3-4e72-9e56-cea3c8474b0c: pr2modules.netlink.exceptions.NetlinkError: (101, 'Network is unreachable')


SUBNETS:

root@deployment:~# openstack subnet list --long
+--------------------------------------+------------------+--------------------------------------+------------------+----------------------------------+------+--------------+-----------------------------+-------------+------------+---------------+---------------+------+
| ID                                   | Name             | Network                              | Subnet           | Project                          | DHCP | Name Servers | Allocation Pools            | Host Routes | IP Version | Gateway       | Service Types | Tags |
+--------------------------------------+------------------+--------------------------------------+------------------+----------------------------------+------+--------------+-----------------------------+-------------+------------+---------------+---------------+------+
| 76d3d8e6-93c4-407a-af8a-b67c06316644 | private-subnet   | 12cd0385-f70d-45f6-afcf-4c835d4039ed | 10.0.0.0/24      | 56648a9748c64ce8a99efab01210ce06 | True |              | 10.0.0.2-10.0.0.254         |             |          4 | 10.0.0.1      |               |      |
| fabe73a6-f17e-437c-96d4-59caa9a7e7f3 | public-subnet | 57bb06e9-a4c1-47b8-94de-a5504d910798 | xxx.xxx.xxx.160/27 | 56648a9748c64ce8a99efab01210ce06 | True | xxx.xxx.xxx.22 | xxx.xxx.xxx.175-xxx.xxx.xxx.190 |             |          4 | xxx.xxx.xxx.161 |               |      |
+--------------------------------------+------------------+--------------------------------------+------------------+----------------------------------+------+--------------+-----------------------------+-------------+------------+---------------+---------------+------+


ROUTER:

root@deployment:~# openstack router list --long
+--------------------------------------+-------------------+--------+-------+----------------------------------+-------------+-------+--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+------+
| ID                                   | Name              | Status | State | Project                          | Distributed | HA    | Routes | External gateway info
                                                                                                                         | Availability zones | Tags |
+--------------------------------------+-------------------+--------+-------+----------------------------------+-------------+-------+--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+------+
| 21ecb828-a0cf-4e8e-a500-9395222efc4b | private-ns-router | ACTIVE | UP    | 56648a9748c64ce8a99efab01210ce06 | False       | False |        | {"network_id": "57bb06e9-a4c1-47b8-94de-a5504d910798", "external_fixed_ips": [{"subnet_id": "fabe73a6-f17e-437c-96d4-59caa9a7e7f3", "ip_address": "xxx.xxx.xxx.190"}], "enable_snat": true} | nova               |      |
+--------------------------------------+-------------------+--------+-------+----------------------------------+-------------+-------+--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+------+

Thank you in advance for the help.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1982882

Title:
  Failed to process compatible router - network is unreachable

Status in neutron:
  New

Bug description:
  I am using kolla-ansible to install OpenStack on 3 servers:

  Controller
  Compute1
  Compute2

  Each server has three network interfaces:
  1. management (10.0.0.0/24)
  2. provider (xxx.xxx.xxx.160/27)
  3. server internet access (xxx.xxx.xxx.160/27)

  After deployment, I am unable to get internet connectivity on any
  instance. The following error is seen in
  /var/log/kolla/neutron/neutron-l3-agent.log after instance creation. I
  am unsure what changes I can make to resolve this. Network and router
  information is below.

  
  LOG:

  2022-07-26 18:29:26.061 18 INFO neutron.agent.l3.agent [-] Starting processing update 4e471427-5bf3-4e72-9e56-cea3c8474b0c, action 3, priority 2, update_id 0f830a94-0c6f-4dff-9063-aabad002173a. Wait time elapsed: 6.087
  2022-07-26 18:29:26.061 18 INFO neutron.agent.l3.agent [-] Starting router update for 4e471427-5bf3-4e72-9e56-cea3c8474b0c, action 3, priority 2, update_id 0f830a94-0c6f-4dff-9063-aabad002173a. Wait time elapsed: 6.087
  2022-07-26 18:29:26.492 18 INFO neutron.agent.linux.interface [-] Device qg-6a95a75b-9b already exists
  2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info [-] (101, 'Network is unreachable'): pr2modules.netlink.exceptions.NetlinkError: (101, 'Network is unreachable')
  2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
  2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/common/utils.py", line 174, in call
  2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info     return func(*args, **kwargs)
  2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 1312, in process
  2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info     self.process_external()
  2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info   File "<decorator-gen-92>", line 2, in process_external
  2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/common/coordination.py", line 78, in _synchronized
  2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info     return f(*a, **k)
  2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 1046, in process_external
  2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info     self._process_external_gateway(ex_gw_port)
  2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 927, in _process_external_gateway
  2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info     self.external_gateway_added(ex_gw_port, interface_name)
  2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 880, in external_gateway_added
  2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info     ex_gw_port, interface_name, self.ns_name, preserve_ips)
  2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 823, in _external_gateway_added
  2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info     ns_name, preserve_ips)
  2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 853, in _external_gateway_settings
  2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info     device.route.add_gateway(ip)
  2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 613, in add_gateway
  2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info     scope=scope)
  2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 651, in add_route
  2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info     table=table, metric=metric, scope=scope, **kwargs)
  2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 1525, in add_ip_route
  2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info     metric=metric, scope=scope, proto=proto, **kwargs)
  2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 272, in _wrap
  2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info     r_call_timeout)
  2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 215, in remote_call
  2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info     raise exc_type(*result[2])
  2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info pr2modules.netlink.exceptions.NetlinkError: (101, 'Network is unreachable')
  2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info
  2022-07-26 18:29:26.977 18 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 4e471427-5bf3-4e72-9e56-cea3c8474b0c: pr2modules.netlink.exceptions.NetlinkError: (101, 'Network is unreachable')

  
  SUBNETS:

  root@deployment:~# openstack subnet list --long
  +--------------------------------------+------------------+--------------------------------------+------------------+----------------------------------+------+--------------+-----------------------------+-------------+------------+---------------+---------------+------+
  | ID                                   | Name             | Network                              | Subnet           | Project                          | DHCP | Name Servers | Allocation Pools            | Host Routes | IP Version | Gateway       | Service Types | Tags |
  +--------------------------------------+------------------+--------------------------------------+------------------+----------------------------------+------+--------------+-----------------------------+-------------+------------+---------------+---------------+------+
  | 76d3d8e6-93c4-407a-af8a-b67c06316644 | private-subnet   | 12cd0385-f70d-45f6-afcf-4c835d4039ed | 10.0.0.0/24      | 56648a9748c64ce8a99efab01210ce06 | True |              | 10.0.0.2-10.0.0.254         |             |          4 | 10.0.0.1      |               |      |
  | fabe73a6-f17e-437c-96d4-59caa9a7e7f3 | public-subnet | 57bb06e9-a4c1-47b8-94de-a5504d910798 | xxx.xxx.xxx.160/27 | 56648a9748c64ce8a99efab01210ce06 | True | xxx.xxx.xxx.22 | xxx.xxx.xxx.175-xxx.xxx.xxx.190 |             |          4 | xxx.xxx.xxx.161 |               |      |
  +--------------------------------------+------------------+--------------------------------------+------------------+----------------------------------+------+--------------+-----------------------------+-------------+------------+---------------+---------------+------+

  
  ROUTER:

  root@deployment:~# openstack router list --long
  +--------------------------------------+-------------------+--------+-------+----------------------------------+-------------+-------+--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+------+
  | ID                                   | Name              | Status | State | Project                          | Distributed | HA    | Routes | External gateway info
                                                                                                                           | Availability zones | Tags |
  +--------------------------------------+-------------------+--------+-------+----------------------------------+-------------+-------+--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+------+
  | 21ecb828-a0cf-4e8e-a500-9395222efc4b | private-ns-router | ACTIVE | UP    | 56648a9748c64ce8a99efab01210ce06 | False       | False |        | {"network_id": "57bb06e9-a4c1-47b8-94de-a5504d910798", "external_fixed_ips": [{"subnet_id": "fabe73a6-f17e-437c-96d4-59caa9a7e7f3", "ip_address": "xxx.xxx.xxx.190"}], "enable_snat": true} | nova               |      |
  +--------------------------------------+-------------------+--------+-------+----------------------------------+-------------+-------+--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+------+

  Thank you in advance for the help.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1982882/+subscriptions