yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1624170] [NEW] Iteration for metering is stopped because metering-agent tried to operate chain in namespace of deleted router

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Kengo Hobo <hobo.kengo@xxxxxxxxxxxxxx>
Date: Fri, 16 Sep 2016 00:50:52 -0000
Reply-to: Bug 1624170 <1624170@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Metering-agent cannot recognize namespace for router is deleted.
Thus, metering-agent sometimes fails to operate chain for metering with following trace
because namespace for router had already deleted, especially when synchronizing information with neutron-server.

I assume that we should simply catch the error and continue the
iteration.

trace in metering-agent.log
=========================
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent [req-4798393c-417b-4af6-bb16-15c7b8cb4b17 - - - - -] Driver neutron.services.meterin
g.drivers.iptables.iptables_driver.IptablesMeteringDriver:update_routers runtime error
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent Traceback (most recent call last):
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent   File "/opt/stack/neutron/neutron/services/metering/agents/metering_agent.py", line
 166, in _invoke_driver
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent     return getattr(self.metering_driver, func_name)(context, meterings)
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent   File "/usr/local/lib/python2.7/dist-packages/oslo_log/helpers.py", line 48, in wra
pper
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent     return method(*args, **kwargs)
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent   File "/opt/stack/neutron/neutron/services/metering/drivers/iptables/iptables_drive
r.py", line 109, in update_routers
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent     self._process_disassociate_metering_label(rm.router)
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent   File "/opt/stack/neutron/neutron/services/metering/drivers/iptables/iptables_drive
r.py", line 257, in _process_disassociate_metering_label
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent     del rm.metering_labels[label_id]
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent   File "/opt/stack/neutron/neutron/services/metering/drivers/iptables/iptables_drive
r.py", line 58, in __exit__
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent     self.im.apply()
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent   File "/opt/stack/neutron/neutron/agent/linux/iptables_manager.py", line 437, in apply
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent     return self._apply()
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent   File "/opt/stack/neutron/neutron/agent/linux/iptables_manager.py", line 445, in _apply
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent     first = self._apply_synchronized()
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent   File "/opt/stack/neutron/neutron/agent/linux/iptables_manager.py", line 480, in _apply_synchronized
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent     save_output = self.execute(args, run_as_root=True)
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent   File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 138, in execute
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent     raise RuntimeError(msg)
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-15e41fb9-44f7-4233-8ff9-12f74f73ade2": No such file or directory
==========================

** Affects: neutron
     Importance: Undecided
     Assignee: Kengo Hobo (hobo-kengo)
         Status: New


** Tags: metering

** Tags added: metering

** Changed in: neutron
     Assignee: (unassigned) => Kengo Hobo (hobo-kengo)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1624170

Title:
  Iteration for metering is stopped because metering-agent tried to
  operate chain in namespace of deleted router

Status in neutron:
  New

Bug description:
  Metering-agent cannot recognize namespace for router is deleted.
  Thus, metering-agent sometimes fails to operate chain for metering with following trace
  because namespace for router had already deleted, especially when synchronizing information with neutron-server.

  I assume that we should simply catch the error and continue the
  iteration.

  trace in metering-agent.log
  =========================
  2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent [req-4798393c-417b-4af6-bb16-15c7b8cb4b17 - - - - -] Driver neutron.services.meterin
  g.drivers.iptables.iptables_driver.IptablesMeteringDriver:update_routers runtime error
  2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent Traceback (most recent call last):
  2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent   File "/opt/stack/neutron/neutron/services/metering/agents/metering_agent.py", line
   166, in _invoke_driver
  2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent     return getattr(self.metering_driver, func_name)(context, meterings)
  2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent   File "/usr/local/lib/python2.7/dist-packages/oslo_log/helpers.py", line 48, in wra
  pper
  2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent     return method(*args, **kwargs)
  2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent   File "/opt/stack/neutron/neutron/services/metering/drivers/iptables/iptables_drive
  r.py", line 109, in update_routers
  2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent     self._process_disassociate_metering_label(rm.router)
  2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent   File "/opt/stack/neutron/neutron/services/metering/drivers/iptables/iptables_drive
  r.py", line 257, in _process_disassociate_metering_label
  2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent     del rm.metering_labels[label_id]
  2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent   File "/opt/stack/neutron/neutron/services/metering/drivers/iptables/iptables_drive
  r.py", line 58, in __exit__
  2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent     self.im.apply()
  2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent   File "/opt/stack/neutron/neutron/agent/linux/iptables_manager.py", line 437, in apply
  2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent     return self._apply()
  2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent   File "/opt/stack/neutron/neutron/agent/linux/iptables_manager.py", line 445, in _apply
  2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent     first = self._apply_synchronized()
  2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent   File "/opt/stack/neutron/neutron/agent/linux/iptables_manager.py", line 480, in _apply_synchronized
  2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent     save_output = self.execute(args, run_as_root=True)
  2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent   File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 138, in execute
  2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent     raise RuntimeError(msg)
  2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-15e41fb9-44f7-4233-8ff9-12f74f73ade2": No such file or directory
  ==========================

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1624170/+subscriptions
Follow ups

[Bug 1624170] Re: Iteration for metering fails because metering-agent tried to operate chain in namespace of deleted router
From: Rodolfo Alonso, 2022-11-28