yahoo-eng-team team mailing list archive

[OpenStack Infra <1566291@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/301685
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1cb43734808eded87210d2957d56b70c514d55c3
Submitter: Jenkins
Branch:    master

commit 1cb43734808eded87210d2957d56b70c514d55c3
Author: Oleg Bondarev <obondarev@xxxxxxxxxxxx>
Date:   Tue Apr 5 16:18:03 2016 +0300

    ADDRESS_SCOPE_MARK_IDS should not be global for L3 agent
    
    Otherwise agent becomes unable to handle more than 1024 routers
    (including deleted routers) and starts failing.
    
    It should be enough to distinguish address scopes inside router namespace,
    so this patch moves ADDRESS_SCOPE_MARK_IDS set to the RouterInfo class.
    
    Closes-Bug: #1566291
    Change-Id: I1e43bb3e68db4db93cc1dfc1383af0311bfb0f2d


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1566291

Title:
  L3 agent: at some point an agent becomes unable to handle new routers

Status in neutron:
  Fix Released

Bug description:
  Following seen in l3 agent logs:

  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent [-] Failed to process compatible router 'e341e0e2-5089-46e9-91f9-2099a156b27f'
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent Traceback (most recent call last):
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 497, in _process_router_update
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 434, in _process_router_if_compatible
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent     self._process_added_router(router)
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 439, in _process_added_router
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent     self._router_added(router['id'], router)
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 340, in _router_added
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent     ri = self._create_router(router_id, router)
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 337, in _create_router
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent     return legacy_router.LegacyRouter(*args, **kwargs)
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 61, in __init__
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent     DEFAULT_ADDRESS_SCOPE: ADDRESS_SCOPE_MARK_IDS.pop()}
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent KeyError: 'pop from an empty set'
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent

  So agent is constantly resyncing (causing load on neutron server) and
  unable to handle new routers.

  I believe that set "ADDRESS_SCOPE_MARK_IDS = set(range(1024, 2048))"
  from router_info.py should not be agent global but it should be
  ADDRESS_SCOPE_MARK_IDS  per router. Or at least need to return values
  back to the set when router is deleted.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1566291/+subscriptions