yahoo-eng-team team mailing list archive

[Ihar Hrachyshka <1632540@xxxxxxxxxxxxxxxxxx>]

** Changed in: neutron
       Status: In Progress => Fix Released

** Tags removed: neutron-proactive-backport-potential

** Tags removed: newton-backport-potential

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1632540

Title:
  l3-agent print the ERROR log in l3 log file continuously ,finally fill
  file space,leading to crash the l3-agent service

Status in neutron:
  Fix Released

Bug description:
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent [req-5d499217-05b6-4a56-a3b7-5681adb53d6c - d2b95803757641b6bc55f6309c12c6e9 - - -] Failed to process compatible router 'da82aeb4-07a4-45ca-ae7a-570aec69df29'
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent Traceback (most recent call last):
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 501, in _process_router_update
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router)
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 438, in _process_router_if_compatible
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self._process_added_router(router)
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 446, in _process_added_router
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent ri.process(self)
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py", line 488, in process
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent super(DvrLocalRouter, self).process(agent)
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_router_base.py", line 30, in process
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent super(DvrRouterBase, self).process(agent)
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 386, in process
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent super(HaRouter, self).process(agent)
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 385, in call
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self.logger(e)
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self.force_reraise()
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb)
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 382, in call
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent return func(*args, **kwargs)
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 964, in process
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self.process_address_scope()
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_edge_router.py", line 239, in process_address_scope
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self.snat_iptables_manager, ports_scopemark)
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self.gen.next()
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py", line 461, in defer_apply
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent raise n_exc.IpTablesApplyException(msg)
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent IpTablesApplyException: Failure applying iptables rules

  for example,this ERROR information will fill l3-agent log file
  continuously until solving the problem ,it will fill the log file
  space.

  because we resyc the ERROR update into the queue when the update is
  not been handle successfully.then the greenthread in l3-agent will
  deal with the update periodicly,so print the log periodicly, but the
  l3 agent has been deal with this update,wo should delete this update.

  we could disable l3-agent in a networknode in ha model, then create
  router,then restart the l3-agent ，we will see this problem。

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1632540/+subscriptions