yahoo-eng-team team mailing list archive

[Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>]

** Changed in: neutron
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1489671

Title:
  Neutron L3 sync_routers logic process all router ports from database
  when even sync for a specific router

Status in neutron:
  Fix Released

Bug description:
  Recreate Steps:
  1) Create multiple routers and allocate each router interface for neutron route ports from different network.
      for example, below, there are 4 routers with each have 4,2,1,2 ports.  (So totally 9 router ports in database)
  [root@controller ~]# neutron router-list
  +--------------------------------------+---------------+-----------------------+-------------+-------+
  | id                                   | name          | external_gateway_info | distributed | ha    |
  +--------------------------------------+---------------+-----------------------+-------------+-------+
  | b2b466d2-1b1a-488d-af92-9d83d1c0f2c0 | routername1   | null                  | False       | False |
  | 919f4312-41d1-47a8-b2b5-dc7f14d3f331 | routername2   | null                  | False       | False |
  | 2854df21-7fe8-4968-a372-3c4a5c3d4ecf | routername3   | null                  | False       | False |
  | daf51173-0084-4881-9ba3-0a9ac80d7d7b | routername4   | null                  | False       | False |
  +--------------------------------------+---------------+-----------------------+-------------+-------+

  [root@controller ~]# neutron router-port-list routername1
  +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
  | id                                   | name | mac_address       | fixed_ips                                                                       |
  +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
  | 6194f014-e7c1-4d0b-835f-3cbf94839b9b |      | fa:16:3e:a9:43:7a | {"subnet_id": "84b1e75e-9ce3-4a85-a9c6-32133fca081d", "ip_address": "77.0.0.1"} |
  | bcac4f23-b74d-4cb3-8bbe-f1d59dff724f |      | fa:16:3e:72:59:a1 | {"subnet_id": "80dc7dfe-d353-4c51-8882-934da8bbbe8b", "ip_address": "77.1.0.1"} |
  | 39bb4b6c-e439-43a3-85f2-cade8bce8d3c |      | fa:16:3e:9a:65:e6 | {"subnet_id": "b54cb217-98b8-41e1-8b6f-fb69d84fcb56", "ip_address": "80.0.0.1"} |
  | 3349d441-4679-4176-9f6f-497d39b37c74 |      | fa:16:3e:eb:43:b5 | {"subnet_id": "8fad7ca7-ae0d-4764-92d9-a5e23e806eba", "ip_address": "81.0.0.1"} |
  +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
  [root@controller ~]# neutron router-port-list routername2
  +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
  | id                                   | name | mac_address       | fixed_ips                                                                       |
  +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
  | 77ac0964-57bf-4ed2-8822-332779e427f2 |      | fa:16:3e:ea:83:f8 | {"subnet_id": "2f07dbf4-9c5c-477c-b992-1d3dd284b987", "ip_address": "95.0.0.1"} |
  | aeeb920e-5c73-45ba-8fe9-f6dafabdab68 |      | fa:16:3e:ee:43:a8 | {"subnet_id": "15c55c9f-2051-4b4d-9628-552b86543e4e", "ip_address": "97.0.0.1"} |
  +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
  [root@controller ~]# neutron router-port-list routername3
  +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
  | id                                   | name | mac_address       | fixed_ips                                                                       |
  +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
  | f792ac7d-0bdd-4dbe-bafb-7822ce388c71 |      | fa:16:3e:fe:b7:f7 | {"subnet_id": "b62990de-0468-4efd-adaf-d421351c6a8b", "ip_address": "66.0.0.1"} |
  +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
  [root@controller ~]# neutron router-port-list routername4
  +--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------+
  | id                                   | name | mac_address       | fixed_ips                                                                         |
  +--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------+
  | d1fded02-d378-4d92-bf0f-31cdc93ab365 |      | fa:16:3e:3f:b1:2a | {"subnet_id": "b55cdcb2-e0e8-4110-8a90-3030930bd3d7", "ip_address": "10.10.10.1"} |
  | 0f8addf1-6c7e-49e9-9f25-f8709718865f |      | fa:16:3e:19:70:84 | {"subnet_id": "089ae5d9-84ca-412c-9842-b20a5a0bb68d", "ip_address": "20.20.20.1"} |
  +--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------+

  2) Create a new router. And bind router to L3 agent.
      neutron router-create newrouter
  [root@controller neutron]# neutron router-create  testroute_1
  +-----------------------+--------------------------------------+
  | Field                 | Value                                |
  +-----------------------+--------------------------------------+
  | admin_state_up        | True                                 |
  | distributed           | False                                |
  | external_gateway_info |                                      |
  | ha                    | False                                |
  | id                    | ef37b0fa-a7f9-4a55-8b05-e9fe966a081d |
  | name                  | testroute_1                          |
  | routes                |                                      |
  | status                | ACTIVE                               |
  | tenant_id             | 4fe64a62e087479a8a5e087441520253     |
  +-----------------------+--------------------------------------+
  [root@controller neutron]# neutron l3-agent-router-add 8c0b936e-b759-4b0d-bf8f-295cb002a206  testroute_1
  Added router testroute_1 to L3 agent

  [root@controller neutron]# neutron router-port-list ef37b0fa-a7f9-4a55-8b05-e9fe966a081d
  [root@controller neutron]#

  Patch the code and find the log in l3_db.py:
  interfaces = [self._core_plugin._make_port_dict(rp.port, None)
                        for rp in qry ]
  LOG.info("###### query router port count: %s for router_ids %s", len(interfaces), router_ids)

  And the log result triggered from above step.
  server.log-20150828:2015-08-27 19:14:56.502 11820 INFO neutron.db.l3_db [req-0c723d30-e7f5-47dc-bb6c-edc36207d18d ] ###### query router port count: 9 for router_ids [u'ef37b0fa-a7f9-4a55-8b05-e9fe966a081d']

  From the log, we can see the DB query method of _get_sync_interfaces
  https://github.com/openstack/neutron/blob/master/neutron/db/l3_db.py#L1264
  will query out ALL the router ports that even not belong to the new create router ID(ef37b0fa-a7f9-4a55-8b05-e9fe966a081d).

  This query will lag down the sync_ports performance when they are
  large number of router ports in the database. And the sync routers can
  be frequently triggered in cases when any new router bind to l3 agent
  or any new router port(interface) is added to an existing router.

  And to mention that the total router list returned from method
  get_sync_data(https://github.com/openstack/neutron/blob/master/neutron/db/l3_db.py#L1278)
  is correct. That is because the self._process_interfaces will filter
  out those unrelated ports
  (https://github.com/openstack/neutron/blob/master/neutron/db/l3_db.py#L1251)
  and mask the problem from discovered.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1489671/+subscriptions