yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #37367
[Bug 1489671] [NEW] Neutron L3 sync_routers logic process all router ports from database when even sync for a specific router
Public bug reported:
Recreate Steps:
1) Create multiple routers and allocate each router interface for neutron route ports from different network.
for example, below, there are 4 routers with each have 4,2,1,2 ports. (So totally 9 router ports in database)
[root@controller ~]# neutron router-list
+--------------------------------------+---------------+-----------------------+-------------+-------+
| id | name | external_gateway_info | distributed | ha |
+--------------------------------------+---------------+-----------------------+-------------+-------+
| b2b466d2-1b1a-488d-af92-9d83d1c0f2c0 | routername1 | null | False | False |
| 919f4312-41d1-47a8-b2b5-dc7f14d3f331 | routername2 | null | False | False |
| 2854df21-7fe8-4968-a372-3c4a5c3d4ecf | routername3 | null | False | False |
| daf51173-0084-4881-9ba3-0a9ac80d7d7b | routername4 | null | False | False |
+--------------------------------------+---------------+-----------------------+-------------+-------+
[root@controller ~]# neutron router-port-list routername1
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
| 6194f014-e7c1-4d0b-835f-3cbf94839b9b | | fa:16:3e:a9:43:7a | {"subnet_id": "84b1e75e-9ce3-4a85-a9c6-32133fca081d", "ip_address": "77.0.0.1"} |
| bcac4f23-b74d-4cb3-8bbe-f1d59dff724f | | fa:16:3e:72:59:a1 | {"subnet_id": "80dc7dfe-d353-4c51-8882-934da8bbbe8b", "ip_address": "77.1.0.1"} |
| 39bb4b6c-e439-43a3-85f2-cade8bce8d3c | | fa:16:3e:9a:65:e6 | {"subnet_id": "b54cb217-98b8-41e1-8b6f-fb69d84fcb56", "ip_address": "80.0.0.1"} |
| 3349d441-4679-4176-9f6f-497d39b37c74 | | fa:16:3e:eb:43:b5 | {"subnet_id": "8fad7ca7-ae0d-4764-92d9-a5e23e806eba", "ip_address": "81.0.0.1"} |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
[root@controller ~]# neutron router-port-list routername2
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
| 77ac0964-57bf-4ed2-8822-332779e427f2 | | fa:16:3e:ea:83:f8 | {"subnet_id": "2f07dbf4-9c5c-477c-b992-1d3dd284b987", "ip_address": "95.0.0.1"} |
| aeeb920e-5c73-45ba-8fe9-f6dafabdab68 | | fa:16:3e:ee:43:a8 | {"subnet_id": "15c55c9f-2051-4b4d-9628-552b86543e4e", "ip_address": "97.0.0.1"} |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
[root@controller ~]# neutron router-port-list routername3
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
| f792ac7d-0bdd-4dbe-bafb-7822ce388c71 | | fa:16:3e:fe:b7:f7 | {"subnet_id": "b62990de-0468-4efd-adaf-d421351c6a8b", "ip_address": "66.0.0.1"} |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
[root@controller ~]# neutron router-port-list routername4
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------+
| d1fded02-d378-4d92-bf0f-31cdc93ab365 | | fa:16:3e:3f:b1:2a | {"subnet_id": "b55cdcb2-e0e8-4110-8a90-3030930bd3d7", "ip_address": "10.10.10.1"} |
| 0f8addf1-6c7e-49e9-9f25-f8709718865f | | fa:16:3e:19:70:84 | {"subnet_id": "089ae5d9-84ca-412c-9842-b20a5a0bb68d", "ip_address": "20.20.20.1"} |
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------+
2) Create a new router. And bind router to L3 agent.
neutron router-create newrouter
[root@controller neutron]# neutron router-create testroute_1
+-----------------------+--------------------------------------+
| Field | Value |
+-----------------------+--------------------------------------+
| admin_state_up | True |
| distributed | False |
| external_gateway_info | |
| ha | False |
| id | ef37b0fa-a7f9-4a55-8b05-e9fe966a081d |
| name | testroute_1 |
| routes | |
| status | ACTIVE |
| tenant_id | 4fe64a62e087479a8a5e087441520253 |
+-----------------------+--------------------------------------+
[root@controller neutron]# neutron l3-agent-router-add 8c0b936e-b759-4b0d-bf8f-295cb002a206 testroute_1
Added router testroute_1 to L3 agent
[root@controller neutron]# neutron router-port-list ef37b0fa-a7f9-4a55-8b05-e9fe966a081d
[root@controller neutron]#
Patch the code and find the log in l3_db.py:
interfaces = [self._core_plugin._make_port_dict(rp.port, None)
for rp in qry ]
LOG.info("###### query router port count: %s for router_ids %s", len(interfaces), router_ids)
And the log result triggered from above step.
server.log-20150828:2015-08-27 19:14:56.502 11820 INFO neutron.db.l3_db [req-0c723d30-e7f5-47dc-bb6c-edc36207d18d ] ###### query router port count: 9 for router_ids [u'ef37b0fa-a7f9-4a55-8b05-e9fe966a081d']
>From the log, we can see the DB query method of _get_sync_interfaces
https://github.com/openstack/neutron/blob/master/neutron/db/l3_db.py#L1264
will query out ALL the port routers that even not belong to the new create router ID.
This query will lag down the sync_ports performance when they are large
number of router ports in the database. And the sync port can be
frequently triggered in cases when any new router bind to l3 agent or
any new router port(interface) is added to an existing router.
And to mention that the total router list returned from method
get_sync_data(https://github.com/openstack/neutron/blob/master/neutron/db/l3_db.py#L1278)
is correct. That is because the self._process_interfaces will filter out
those unrelated ports
(https://github.com/openstack/neutron/blob/master/neutron/db/l3_db.py#L1251)
and mask the problem from discovered.
When new router is created and add to a L3 agent, neutron server will
sync ports for the new router.
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1489671
Title:
Neutron L3 sync_routers logic process all router ports from database
when even sync for a specific router
Status in neutron:
New
Bug description:
Recreate Steps:
1) Create multiple routers and allocate each router interface for neutron route ports from different network.
for example, below, there are 4 routers with each have 4,2,1,2 ports. (So totally 9 router ports in database)
[root@controller ~]# neutron router-list
+--------------------------------------+---------------+-----------------------+-------------+-------+
| id | name | external_gateway_info | distributed | ha |
+--------------------------------------+---------------+-----------------------+-------------+-------+
| b2b466d2-1b1a-488d-af92-9d83d1c0f2c0 | routername1 | null | False | False |
| 919f4312-41d1-47a8-b2b5-dc7f14d3f331 | routername2 | null | False | False |
| 2854df21-7fe8-4968-a372-3c4a5c3d4ecf | routername3 | null | False | False |
| daf51173-0084-4881-9ba3-0a9ac80d7d7b | routername4 | null | False | False |
+--------------------------------------+---------------+-----------------------+-------------+-------+
[root@controller ~]# neutron router-port-list routername1
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
| 6194f014-e7c1-4d0b-835f-3cbf94839b9b | | fa:16:3e:a9:43:7a | {"subnet_id": "84b1e75e-9ce3-4a85-a9c6-32133fca081d", "ip_address": "77.0.0.1"} |
| bcac4f23-b74d-4cb3-8bbe-f1d59dff724f | | fa:16:3e:72:59:a1 | {"subnet_id": "80dc7dfe-d353-4c51-8882-934da8bbbe8b", "ip_address": "77.1.0.1"} |
| 39bb4b6c-e439-43a3-85f2-cade8bce8d3c | | fa:16:3e:9a:65:e6 | {"subnet_id": "b54cb217-98b8-41e1-8b6f-fb69d84fcb56", "ip_address": "80.0.0.1"} |
| 3349d441-4679-4176-9f6f-497d39b37c74 | | fa:16:3e:eb:43:b5 | {"subnet_id": "8fad7ca7-ae0d-4764-92d9-a5e23e806eba", "ip_address": "81.0.0.1"} |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
[root@controller ~]# neutron router-port-list routername2
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
| 77ac0964-57bf-4ed2-8822-332779e427f2 | | fa:16:3e:ea:83:f8 | {"subnet_id": "2f07dbf4-9c5c-477c-b992-1d3dd284b987", "ip_address": "95.0.0.1"} |
| aeeb920e-5c73-45ba-8fe9-f6dafabdab68 | | fa:16:3e:ee:43:a8 | {"subnet_id": "15c55c9f-2051-4b4d-9628-552b86543e4e", "ip_address": "97.0.0.1"} |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
[root@controller ~]# neutron router-port-list routername3
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
| f792ac7d-0bdd-4dbe-bafb-7822ce388c71 | | fa:16:3e:fe:b7:f7 | {"subnet_id": "b62990de-0468-4efd-adaf-d421351c6a8b", "ip_address": "66.0.0.1"} |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
[root@controller ~]# neutron router-port-list routername4
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------+
| d1fded02-d378-4d92-bf0f-31cdc93ab365 | | fa:16:3e:3f:b1:2a | {"subnet_id": "b55cdcb2-e0e8-4110-8a90-3030930bd3d7", "ip_address": "10.10.10.1"} |
| 0f8addf1-6c7e-49e9-9f25-f8709718865f | | fa:16:3e:19:70:84 | {"subnet_id": "089ae5d9-84ca-412c-9842-b20a5a0bb68d", "ip_address": "20.20.20.1"} |
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------+
2) Create a new router. And bind router to L3 agent.
neutron router-create newrouter
[root@controller neutron]# neutron router-create testroute_1
+-----------------------+--------------------------------------+
| Field | Value |
+-----------------------+--------------------------------------+
| admin_state_up | True |
| distributed | False |
| external_gateway_info | |
| ha | False |
| id | ef37b0fa-a7f9-4a55-8b05-e9fe966a081d |
| name | testroute_1 |
| routes | |
| status | ACTIVE |
| tenant_id | 4fe64a62e087479a8a5e087441520253 |
+-----------------------+--------------------------------------+
[root@controller neutron]# neutron l3-agent-router-add 8c0b936e-b759-4b0d-bf8f-295cb002a206 testroute_1
Added router testroute_1 to L3 agent
[root@controller neutron]# neutron router-port-list ef37b0fa-a7f9-4a55-8b05-e9fe966a081d
[root@controller neutron]#
Patch the code and find the log in l3_db.py:
interfaces = [self._core_plugin._make_port_dict(rp.port, None)
for rp in qry ]
LOG.info("###### query router port count: %s for router_ids %s", len(interfaces), router_ids)
And the log result triggered from above step.
server.log-20150828:2015-08-27 19:14:56.502 11820 INFO neutron.db.l3_db [req-0c723d30-e7f5-47dc-bb6c-edc36207d18d ] ###### query router port count: 9 for router_ids [u'ef37b0fa-a7f9-4a55-8b05-e9fe966a081d']
From the log, we can see the DB query method of _get_sync_interfaces
https://github.com/openstack/neutron/blob/master/neutron/db/l3_db.py#L1264
will query out ALL the port routers that even not belong to the new create router ID.
This query will lag down the sync_ports performance when they are
large number of router ports in the database. And the sync port can be
frequently triggered in cases when any new router bind to l3 agent or
any new router port(interface) is added to an existing router.
And to mention that the total router list returned from method
get_sync_data(https://github.com/openstack/neutron/blob/master/neutron/db/l3_db.py#L1278)
is correct. That is because the self._process_interfaces will filter
out those unrelated ports
(https://github.com/openstack/neutron/blob/master/neutron/db/l3_db.py#L1251)
and mask the problem from discovered.
When new router is created and add to a L3 agent, neutron server will
sync ports for the new router.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1489671/+subscriptions
Follow ups
