yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1632540] [NEW] l3-agent print the ERROR log in l3 log file continuously , finally fill file space, leading to crash the l3-agent service

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: zhichao zhu <rtmdk@xxxxxxx>
Date: Wed, 12 Oct 2016 02:54:46 -0000
Reply-to: Bug 1632540 <1632540@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent [req-5d499217-05b6-4a56-a3b7-5681adb53d6c - d2b95803757641b6bc55f6309c12c6e9 - - -] Failed to process compatible router 'da82aeb4-07a4-45ca-ae7a-570aec69df29'
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 501, in _process_router_update
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router)
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 438, in _process_router_if_compatible
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self._process_added_router(router)
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 446, in _process_added_router
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent ri.process(self)
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py", line 488, in process
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent super(DvrLocalRouter, self).process(agent)
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_router_base.py", line 30, in process
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent super(DvrRouterBase, self).process(agent)
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 386, in process
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent super(HaRouter, self).process(agent)
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 385, in call
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self.logger(e)
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self.force_reraise()
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb)
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 382, in call
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent return func(*args, **kwargs)
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 964, in process
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self.process_address_scope()
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_edge_router.py", line 239, in process_address_scope
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self.snat_iptables_manager, ports_scopemark)
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self.gen.next()
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py", line 461, in defer_apply
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent raise n_exc.IpTablesApplyException(msg)
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent IpTablesApplyException: Failure applying iptables rules

this ERROR information will fill l3-agent log file continuously until
solving the problem ,it will fill the file space.

** Affects: neutron
     Importance: Undecided
     Assignee: zhichao zhu (rtmdk)
         Status: New


** Tags: l3-agent

** Changed in: neutron
     Assignee: (unassigned) => zhichao zhu (rtmdk)

** Tags removed: l3-dvr
** Tags added: l3-agent

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1632540

Title:
  l3-agent print the ERROR log in l3 log file continuously ,finally fill
  file space,leading to crash the l3-agent service

Status in neutron:
  New

Bug description:
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent [req-5d499217-05b6-4a56-a3b7-5681adb53d6c - d2b95803757641b6bc55f6309c12c6e9 - - -] Failed to process compatible router 'da82aeb4-07a4-45ca-ae7a-570aec69df29'
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent Traceback (most recent call last):
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 501, in _process_router_update
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router)
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 438, in _process_router_if_compatible
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self._process_added_router(router)
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 446, in _process_added_router
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent ri.process(self)
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py", line 488, in process
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent super(DvrLocalRouter, self).process(agent)
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_router_base.py", line 30, in process
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent super(DvrRouterBase, self).process(agent)
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 386, in process
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent super(HaRouter, self).process(agent)
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 385, in call
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self.logger(e)
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self.force_reraise()
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb)
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 382, in call
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent return func(*args, **kwargs)
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 964, in process
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self.process_address_scope()
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_edge_router.py", line 239, in process_address_scope
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self.snat_iptables_manager, ports_scopemark)
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self.gen.next()
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py", line 461, in defer_apply
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent raise n_exc.IpTablesApplyException(msg)
  2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent IpTablesApplyException: Failure applying iptables rules

  this ERROR information will fill l3-agent log file continuously until
  solving the problem ,it will fill the file space.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1632540/+subscriptions
Follow ups

[Bug 1632540] Re: l3-agent print the ERROR log in l3 log file continuously , finally fill file space, leading to crash the l3-agent service
From: Ihar Hrachyshka, 2017-06-14