← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1489671] [NEW] Neutron L3 sync_routers logic process all router ports from database when even sync for a specific router

 

Public bug reported:

Recreate Steps:
1) Create multiple routers and allocate each router interface for neutron route ports from different network.
    for example, below, there are 4 routers with each have 4,2,1,2 ports.  (So totally 9 router ports in database)
[root@controller ~]# neutron router-list
+--------------------------------------+---------------+-----------------------+-------------+-------+
| id                                   | name          | external_gateway_info | distributed | ha    |
+--------------------------------------+---------------+-----------------------+-------------+-------+
| b2b466d2-1b1a-488d-af92-9d83d1c0f2c0 | routername1   | null                  | False       | False |
| 919f4312-41d1-47a8-b2b5-dc7f14d3f331 | routername2   | null                  | False       | False |
| 2854df21-7fe8-4968-a372-3c4a5c3d4ecf | routername3   | null                  | False       | False |
| daf51173-0084-4881-9ba3-0a9ac80d7d7b | routername4   | null                  | False       | False |
+--------------------------------------+---------------+-----------------------+-------------+-------+

[root@controller ~]# neutron router-port-list routername1
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                       |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
| 6194f014-e7c1-4d0b-835f-3cbf94839b9b |      | fa:16:3e:a9:43:7a | {"subnet_id": "84b1e75e-9ce3-4a85-a9c6-32133fca081d", "ip_address": "77.0.0.1"} |
| bcac4f23-b74d-4cb3-8bbe-f1d59dff724f |      | fa:16:3e:72:59:a1 | {"subnet_id": "80dc7dfe-d353-4c51-8882-934da8bbbe8b", "ip_address": "77.1.0.1"} |
| 39bb4b6c-e439-43a3-85f2-cade8bce8d3c |      | fa:16:3e:9a:65:e6 | {"subnet_id": "b54cb217-98b8-41e1-8b6f-fb69d84fcb56", "ip_address": "80.0.0.1"} |
| 3349d441-4679-4176-9f6f-497d39b37c74 |      | fa:16:3e:eb:43:b5 | {"subnet_id": "8fad7ca7-ae0d-4764-92d9-a5e23e806eba", "ip_address": "81.0.0.1"} |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
[root@controller ~]# neutron router-port-list routername2
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                       |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
| 77ac0964-57bf-4ed2-8822-332779e427f2 |      | fa:16:3e:ea:83:f8 | {"subnet_id": "2f07dbf4-9c5c-477c-b992-1d3dd284b987", "ip_address": "95.0.0.1"} |
| aeeb920e-5c73-45ba-8fe9-f6dafabdab68 |      | fa:16:3e:ee:43:a8 | {"subnet_id": "15c55c9f-2051-4b4d-9628-552b86543e4e", "ip_address": "97.0.0.1"} |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
[root@controller ~]# neutron router-port-list routername3
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                       |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
| f792ac7d-0bdd-4dbe-bafb-7822ce388c71 |      | fa:16:3e:fe:b7:f7 | {"subnet_id": "b62990de-0468-4efd-adaf-d421351c6a8b", "ip_address": "66.0.0.1"} |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
[root@controller ~]# neutron router-port-list routername4
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                         |
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------+
| d1fded02-d378-4d92-bf0f-31cdc93ab365 |      | fa:16:3e:3f:b1:2a | {"subnet_id": "b55cdcb2-e0e8-4110-8a90-3030930bd3d7", "ip_address": "10.10.10.1"} |
| 0f8addf1-6c7e-49e9-9f25-f8709718865f |      | fa:16:3e:19:70:84 | {"subnet_id": "089ae5d9-84ca-412c-9842-b20a5a0bb68d", "ip_address": "20.20.20.1"} |
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------+


2) Create a new router. And bind router to L3 agent. 
    neutron router-create newrouter
[root@controller neutron]# neutron router-create  testroute_1 
+-----------------------+--------------------------------------+
| Field                 | Value                                |
+-----------------------+--------------------------------------+
| admin_state_up        | True                                 |
| distributed           | False                                |
| external_gateway_info |                                      |
| ha                    | False                                |
| id                    | ef37b0fa-a7f9-4a55-8b05-e9fe966a081d |
| name                  | testroute_1                          |
| routes                |                                      |
| status                | ACTIVE                               |
| tenant_id             | 4fe64a62e087479a8a5e087441520253     |
+-----------------------+--------------------------------------+
[root@controller neutron]# neutron l3-agent-router-add 8c0b936e-b759-4b0d-bf8f-295cb002a206  testroute_1  
Added router testroute_1 to L3 agent

[root@controller neutron]# neutron router-port-list ef37b0fa-a7f9-4a55-8b05-e9fe966a081d
[root@controller neutron]#

Patch the code and find the log in l3_db.py:
interfaces = [self._core_plugin._make_port_dict(rp.port, None)
                      for rp in qry ]
LOG.info("###### query router port count: %s for router_ids %s", len(interfaces), router_ids)

And the log result triggered from above step. 
server.log-20150828:2015-08-27 19:14:56.502 11820 INFO neutron.db.l3_db [req-0c723d30-e7f5-47dc-bb6c-edc36207d18d ] ###### query router port count: 9 for router_ids [u'ef37b0fa-a7f9-4a55-8b05-e9fe966a081d']


>From the log, we can see the DB query method of _get_sync_interfaces 
https://github.com/openstack/neutron/blob/master/neutron/db/l3_db.py#L1264
will query out ALL the port routers that even not belong to the new create router ID.  

This query will lag down the sync_ports performance when they are large
number of router ports in the database. And the sync port can be
frequently triggered in cases when any new router bind to l3 agent or
any new router port(interface) is added to an existing router.

And to mention that the total router list returned from method
get_sync_data(https://github.com/openstack/neutron/blob/master/neutron/db/l3_db.py#L1278)
is correct. That is because the self._process_interfaces will filter out
those unrelated ports
(https://github.com/openstack/neutron/blob/master/neutron/db/l3_db.py#L1251)
and mask the problem from discovered.


When new router is created and add to a L3 agent, neutron server will
sync ports for the new router.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1489671

Title:
  Neutron L3 sync_routers logic process all router ports from database
  when even sync for a specific router

Status in neutron:
  New

Bug description:
  Recreate Steps:
  1) Create multiple routers and allocate each router interface for neutron route ports from different network.
      for example, below, there are 4 routers with each have 4,2,1,2 ports.  (So totally 9 router ports in database)
  [root@controller ~]# neutron router-list
  +--------------------------------------+---------------+-----------------------+-------------+-------+
  | id                                   | name          | external_gateway_info | distributed | ha    |
  +--------------------------------------+---------------+-----------------------+-------------+-------+
  | b2b466d2-1b1a-488d-af92-9d83d1c0f2c0 | routername1   | null                  | False       | False |
  | 919f4312-41d1-47a8-b2b5-dc7f14d3f331 | routername2   | null                  | False       | False |
  | 2854df21-7fe8-4968-a372-3c4a5c3d4ecf | routername3   | null                  | False       | False |
  | daf51173-0084-4881-9ba3-0a9ac80d7d7b | routername4   | null                  | False       | False |
  +--------------------------------------+---------------+-----------------------+-------------+-------+

  [root@controller ~]# neutron router-port-list routername1
  +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
  | id                                   | name | mac_address       | fixed_ips                                                                       |
  +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
  | 6194f014-e7c1-4d0b-835f-3cbf94839b9b |      | fa:16:3e:a9:43:7a | {"subnet_id": "84b1e75e-9ce3-4a85-a9c6-32133fca081d", "ip_address": "77.0.0.1"} |
  | bcac4f23-b74d-4cb3-8bbe-f1d59dff724f |      | fa:16:3e:72:59:a1 | {"subnet_id": "80dc7dfe-d353-4c51-8882-934da8bbbe8b", "ip_address": "77.1.0.1"} |
  | 39bb4b6c-e439-43a3-85f2-cade8bce8d3c |      | fa:16:3e:9a:65:e6 | {"subnet_id": "b54cb217-98b8-41e1-8b6f-fb69d84fcb56", "ip_address": "80.0.0.1"} |
  | 3349d441-4679-4176-9f6f-497d39b37c74 |      | fa:16:3e:eb:43:b5 | {"subnet_id": "8fad7ca7-ae0d-4764-92d9-a5e23e806eba", "ip_address": "81.0.0.1"} |
  +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
  [root@controller ~]# neutron router-port-list routername2
  +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
  | id                                   | name | mac_address       | fixed_ips                                                                       |
  +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
  | 77ac0964-57bf-4ed2-8822-332779e427f2 |      | fa:16:3e:ea:83:f8 | {"subnet_id": "2f07dbf4-9c5c-477c-b992-1d3dd284b987", "ip_address": "95.0.0.1"} |
  | aeeb920e-5c73-45ba-8fe9-f6dafabdab68 |      | fa:16:3e:ee:43:a8 | {"subnet_id": "15c55c9f-2051-4b4d-9628-552b86543e4e", "ip_address": "97.0.0.1"} |
  +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
  [root@controller ~]# neutron router-port-list routername3
  +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
  | id                                   | name | mac_address       | fixed_ips                                                                       |
  +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
  | f792ac7d-0bdd-4dbe-bafb-7822ce388c71 |      | fa:16:3e:fe:b7:f7 | {"subnet_id": "b62990de-0468-4efd-adaf-d421351c6a8b", "ip_address": "66.0.0.1"} |
  +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
  [root@controller ~]# neutron router-port-list routername4
  +--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------+
  | id                                   | name | mac_address       | fixed_ips                                                                         |
  +--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------+
  | d1fded02-d378-4d92-bf0f-31cdc93ab365 |      | fa:16:3e:3f:b1:2a | {"subnet_id": "b55cdcb2-e0e8-4110-8a90-3030930bd3d7", "ip_address": "10.10.10.1"} |
  | 0f8addf1-6c7e-49e9-9f25-f8709718865f |      | fa:16:3e:19:70:84 | {"subnet_id": "089ae5d9-84ca-412c-9842-b20a5a0bb68d", "ip_address": "20.20.20.1"} |
  +--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------+

  
  2) Create a new router. And bind router to L3 agent. 
      neutron router-create newrouter
  [root@controller neutron]# neutron router-create  testroute_1 
  +-----------------------+--------------------------------------+
  | Field                 | Value                                |
  +-----------------------+--------------------------------------+
  | admin_state_up        | True                                 |
  | distributed           | False                                |
  | external_gateway_info |                                      |
  | ha                    | False                                |
  | id                    | ef37b0fa-a7f9-4a55-8b05-e9fe966a081d |
  | name                  | testroute_1                          |
  | routes                |                                      |
  | status                | ACTIVE                               |
  | tenant_id             | 4fe64a62e087479a8a5e087441520253     |
  +-----------------------+--------------------------------------+
  [root@controller neutron]# neutron l3-agent-router-add 8c0b936e-b759-4b0d-bf8f-295cb002a206  testroute_1  
  Added router testroute_1 to L3 agent

  [root@controller neutron]# neutron router-port-list ef37b0fa-a7f9-4a55-8b05-e9fe966a081d
  [root@controller neutron]#

  Patch the code and find the log in l3_db.py:
  interfaces = [self._core_plugin._make_port_dict(rp.port, None)
                        for rp in qry ]
  LOG.info("###### query router port count: %s for router_ids %s", len(interfaces), router_ids)

  And the log result triggered from above step. 
  server.log-20150828:2015-08-27 19:14:56.502 11820 INFO neutron.db.l3_db [req-0c723d30-e7f5-47dc-bb6c-edc36207d18d ] ###### query router port count: 9 for router_ids [u'ef37b0fa-a7f9-4a55-8b05-e9fe966a081d']

  
  From the log, we can see the DB query method of _get_sync_interfaces 
  https://github.com/openstack/neutron/blob/master/neutron/db/l3_db.py#L1264
  will query out ALL the port routers that even not belong to the new create router ID.  

  This query will lag down the sync_ports performance when they are
  large number of router ports in the database. And the sync port can be
  frequently triggered in cases when any new router bind to l3 agent or
  any new router port(interface) is added to an existing router.

  And to mention that the total router list returned from method
  get_sync_data(https://github.com/openstack/neutron/blob/master/neutron/db/l3_db.py#L1278)
  is correct. That is because the self._process_interfaces will filter
  out those unrelated ports
  (https://github.com/openstack/neutron/blob/master/neutron/db/l3_db.py#L1251)
  and mask the problem from discovered.


  When new router is created and add to a L3 agent, neutron server will
  sync ports for the new router.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1489671/+subscriptions


Follow ups