← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #48994

[Bug 1566291] [NEW] L3 agent: at some point an agent becomes unable to handle new routers

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Following seen in l3 agent logs:

2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent [-] Failed to process compatible router 'e341e0e2-5089-46e9-91f9-2099a156b27f'
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 497, in _process_router_update
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 434, in _process_router_if_compatible
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent     self._process_added_router(router)
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 439, in _process_added_router
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent     self._router_added(router['id'], router)
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 340, in _router_added
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent     ri = self._create_router(router_id, router)
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 337, in _create_router
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent     return legacy_router.LegacyRouter(*args, **kwargs)
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 61, in __init__
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent     DEFAULT_ADDRESS_SCOPE: ADDRESS_SCOPE_MARK_IDS.pop()}
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent KeyError: 'pop from an empty set'
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent
2016-04-05 09:30:09.034 24216 DEBUG neutron.agent.l3.agent [-] Starting router update for e341e0e2-5089-46e9-91f9-2099a156b27f, action None, priority 1 _process_router_update /usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py:463
2016-04-05 09:30:09.035 24216 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: 6295fbe9cf2040d79c68f5c5f8b1e963 exchange 'neutron' topic 'q-l3-plugin' _send /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:454
2016-04-05 09:30:09.417 24216 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 6295fbe9cf2040d79c68f5c5f8b1e963 __call__ /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:302
2016-04-05 09:30:09.418 24216 ERROR neutron.agent.l3.agent [-] Failed to process compatible router 'e341e0e2-5089-46e9-91f9-2099a156b27f'

So agent is constantly resyncing (causing load on neutron server) and
unable to handle new routers.

I believe that set "ADDRESS_SCOPE_MARK_IDS = set(range(1024, 2048))"
from router_info.py should not be agent global but it should be
ADDRESS_SCOPE_MARK_IDS  per router. Or at least need to return values
back to the set when router is deleted.

** Affects: neutron
     Importance: Undecided
     Assignee: Oleg Bondarev (obondarev)
         Status: New


** Tags: l3-ipam-dhcp

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1566291

Title:
  L3 agent: at some point an agent becomes unable to handle new routers

Status in neutron:
  New

Bug description:
  Following seen in l3 agent logs:

  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent [-] Failed to process compatible router 'e341e0e2-5089-46e9-91f9-2099a156b27f'
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent Traceback (most recent call last):
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 497, in _process_router_update
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 434, in _process_router_if_compatible
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent     self._process_added_router(router)
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 439, in _process_added_router
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent     self._router_added(router['id'], router)
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 340, in _router_added
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent     ri = self._create_router(router_id, router)
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 337, in _create_router
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent     return legacy_router.LegacyRouter(*args, **kwargs)
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 61, in __init__
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent     DEFAULT_ADDRESS_SCOPE: ADDRESS_SCOPE_MARK_IDS.pop()}
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent KeyError: 'pop from an empty set'
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent
  2016-04-05 09:30:09.034 24216 DEBUG neutron.agent.l3.agent [-] Starting router update for e341e0e2-5089-46e9-91f9-2099a156b27f, action None, priority 1 _process_router_update /usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py:463
  2016-04-05 09:30:09.035 24216 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: 6295fbe9cf2040d79c68f5c5f8b1e963 exchange 'neutron' topic 'q-l3-plugin' _send /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:454
  2016-04-05 09:30:09.417 24216 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 6295fbe9cf2040d79c68f5c5f8b1e963 __call__ /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:302
  2016-04-05 09:30:09.418 24216 ERROR neutron.agent.l3.agent [-] Failed to process compatible router 'e341e0e2-5089-46e9-91f9-2099a156b27f'

  So agent is constantly resyncing (causing load on neutron server) and
  unable to handle new routers.

  I believe that set "ADDRESS_SCOPE_MARK_IDS = set(range(1024, 2048))"
  from router_info.py should not be agent global but it should be
  ADDRESS_SCOPE_MARK_IDS  per router. Or at least need to return values
  back to the set when router is deleted.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1566291/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next