yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1580548] [NEW] Metering-agent's routers information is inconsistent with neutron-server

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: HuntXu <1580548@xxxxxxxxxxxxxxxxxx>
Date: Wed, 11 May 2016 11:12:13 -0000
Reply-to: Bug 1580548 <1580548@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

With the current implement, under certain curcumstance, router
information(including internal-stored data, iptables settings) kept and
set up by metering-agent could be inconsistent with what the neutron-
server suppose it to be. And that cause the metering-agent to run into a
problematic status.

Version:
Neutron master commit 7327e8c21(Merge "Fix update target tenant RBAC external path")

Steps to reproduce:
1. Create internal net, subnet, router. Set external gateway for router, add interface to the router for the created subnet.
2. Create neutron metering label and rule.
3. Make sure the created router is added to an active l3-agent.
4. Create another router and set its external gateway.
5. Use 'neutron l3-agent-router-remove' to remove the first router from the l3 agent.
6. Use 'neutron l3-agent-router-add' to add the first router back to the l3 agent by which it was hosted.


(In the following logs, the first created router's id is 0dd35572-735a-4ecd-8429-e047ed385100, the second one is 392820a7-3908-49eb-98a3-967f1638432f)

Logs in metering-agent.log after step 4:
2016-05-11 18:32:49.915 26409 DEBUG neutron.agent.linux.utils [req-0515c6f8-cbd1-47d4-a45e-f89e6b6e97b4 - - - - -] Running command: ['sudo', 'ip', 'netns', 'exec', 'qrouter-392820a7-3908-49eb-98a3-967f1638432f', 'iptables-save'] create_process /usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:82
2016-05-11 18:32:49.926 26409 ERROR neutron.agent.linux.utils [req-0515c6f8-cbd1-47d4-a45e-f89e6b6e97b4 - - - - -] Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-392820a7-3908-49eb-98a3-967f1638432f": No such file or directory

2016-05-11 18:32:49.926 26409 DEBUG oslo_concurrency.lockutils [req-0515c6f8-cbd1-47d4-a45e-f89e6b6e97b4 - - - - -] Releasing semaphore "iptables-qrouter-392820a7-3908-49eb-98a3-967f1638432f" lock /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:225
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent [req-0515c6f8-cbd1-47d4-a45e-f89e6b6e97b4 - - - - -] Driver neutron.services.metering.drivers.iptables.iptables_driver.IptablesMeteringDriver:update_routers runtime error
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent Traceback (most recent call last):
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent   File "/usr/local/lib/python2.7/dist-packages/neutron/services/metering/agents/metering_agent.py", line 177, in _invoke_driver
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent     return getattr(self.metering_driver, func_name)(context, meterings)
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent   File "/usr/local/lib/python2.7/dist-packages/oslo_log/helpers.py", line 48, in wrapper
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent     return method(*args, **kwargs)
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent   File "/usr/local/lib/python2.7/dist-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 125, in update_routers
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent     self._process_associate_metering_label(router)
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent   File "/usr/local/lib/python2.7/dist-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 228, in _process_associate_metering_label
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent     rm.metering_labels[label_id] = label
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent   File "/usr/local/lib/python2.7/dist-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 58, in __exit__
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent     self.im.apply()
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent   File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/iptables_manager.py", line 431, in apply
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent     return self._apply()
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent   File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/iptables_manager.py", line 439, in _apply
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent     return self._apply_synchronized()
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent   File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/iptables_manager.py", line 465, in _apply_synchronized
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent     save_output = self.execute(args, run_as_root=True)
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent   File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in execute
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent     raise RuntimeError(msg)
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-392820a7-3908-49eb-98a3-967f1638432f": No such file or directory
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent
...Skipping...
2016-05-11 18:33:06.776 26409 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'ip', 'netns', 'exec', 'qrouter-392820a7-3908-49eb-98a3-967f1638432f', 'iptables', '-t', 'filter', '-L', 'neutron-meter-l-1811c9b9-f9b', '-n', '-v', '-x', '-Z'] create_process /usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:82
2016-05-11 18:33:06.787 26409 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-392820a7-3908-49eb-98a3-967f1638432f": No such file or directory

2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver [-] Failed to get traffic counters, router: {u'status': u'ACTIVE', u'name': u'router-two', u'gw_port_id': u'729fba34-4aec-415d-bef8-b93b34e69536', u'admin_state_up': False, u'tenant_id': u'e60fad96858e431cb3da4f0c1ab2d733', u'_metering_labels': [{u'rules': [{u'remote_ip_prefix': u'192.168.88.0/24', u'direction': u'egress', u'metering_label_id': u'1811c9b9-f9b0-40a6-839a-250b20674860', u'id': u'007e3f9e-4e7c-439d-a606-db6ad0ae11ca', u'excluded': False}], u'id': u'1811c9b9-f9b0-40a6-839a-250b20674860'}, {u'rules': [{u'remote_ip_prefix': u'192.168.88.0/24', u'direction': u'ingress', u'metering_label_id': u'd05ce8fb-c013-4ae4-bac5-de1be3f65331', u'id': u'6c580a70-5dec-4225-ade8-44e03fdc9f1b', u'excluded': False}], u'id': u'd05ce8fb-c013-4ae4-bac5-de1be3f65331'}], u'id': u'392820a7-3908-49eb-98a3-967f1638432f'}
2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver Traceback (most recent call last):
2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver   File "/usr/local/lib/python2.7/dist-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 359, in get_traffic_counters
2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver     chain, wrap=False, zero=True)
2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver   File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/iptables_manager.py", line 669, in get_traffic_counters
2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver     current_table = self.execute(args, run_as_root=True)
2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver   File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in execute
2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver     raise RuntimeError(msg)
2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-392820a7-3908-49eb-98a3-967f1638432f": No such file or directory
2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver
2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver
2016-05-11 18:33:06.790 26409 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'ip', 'netns', 'exec', 'qrouter-392820a7-3908-49eb-98a3-967f1638432f', 'iptables', '-t', 'filter', '-L', 'neutron-meter-l-d05ce8fb-c01', '-n', '-v', '-x', '-Z'] create_process /usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:82


Logs in metering-agent.log after step 5:
2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver [-] Failed to get traffic counters, router: {u'status': u'ACTIVE', u'name': u'router-two', u'gw_port_id': u'729fba34-4aec-415d-bef8-b93b34e69536', u'admin_state_up': False, u'tenant_id': u'e60fad96858e431cb3da4f0c1ab2d733', u'_metering_labels': [{u'rules': [{u'remote_ip_prefix': u'192.168.88.0/24', u'direction': u'egress', u'metering_label_id': u'1811c9b9-f9b0-40a6-839a-250b20674860', u'id': u'007e3f9e-4e7c-439d-a606-db6ad0ae11ca', u'excluded': False}], u'id': u'1811c9b9-f9b0-40a6-839a-250b20674860'}, {u'rules': [{u'remote_ip_prefix': u'192.168.88.0/24', u'direction': u'ingress', u'metering_label_id': u'd05ce8fb-c013-4ae4-bac5-de1be3f65331', u'id': u'6c580a70-5dec-4225-ade8-44e03fdc9f1b', u'excluded': False}], u'id': u'd05ce8fb-c013-4ae4-bac5-de1be3f65331'}], u'id': u'392820a7-3908-49eb-98a3-967f1638432f'}
2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver Traceback (most recent call last):
2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver   File "/usr/local/lib/python2.7/dist-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 359, in get_traffic_counters
2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver     chain, wrap=False, zero=True)
2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver   File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/iptables_manager.py", line 669, in get_traffic_counters
2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver     current_table = self.execute(args, run_as_root=True)
2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver   File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in execute
2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver     raise RuntimeError(msg)
2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-392820a7-3908-49eb-98a3-967f1638432f": No such file or directory
2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver 
2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver 
2016-05-11 18:33:36.807 26409 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'ip', 'netns', 'exec', 'qrouter-0dd35572-735a-4ecd-8429-e047ed385100', 'iptables', '-t', 'filter', '-L', 'neutron-meter-l-1811c9b9-f9b', '-n', '-v', '-x', '-Z'] create_process /usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:82
2016-05-11 18:33:36.818 26409 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-0dd35572-735a-4ecd-8429-e047ed385100": No such file or directory


Logs in metering-agent.log after step 6:
2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver [-] Failed to get traffic counters, router: {u'status': u'ACTIVE', u'name': u'router-one', u'gw_port_id': u'a4137128-27d6-43d6-b04d-a5df8ea62b02', u'admin_state_up': True, u'tenant_id': u'e60fad96858e431cb3da4f0c1ab2d733', u'_metering_labels': [{u'rules': [{u'remote_ip_prefix': u'192.168.88.0/24', u'direction': u'egress', u'metering_label_id': u'1811c9b9-f9b0-40a6-839a-250b20674860', u'id': u'007e3f9e-4e7c-439d-a606-db6ad0ae11ca', u'excluded': False}], u'id': u'1811c9b9-f9b0-40a6-839a-250b20674860'}, {u'rules': [{u'remote_ip_prefix': u'192.168.88.0/24', u'direction': u'ingress', u'metering_label_id': u'd05ce8fb-c013-4ae4-bac5-de1be3f65331', u'id': u'6c580a70-5dec-4225-ade8-44e03fdc9f1b', u'excluded': False}], u'id': u'd05ce8fb-c013-4ae4-bac5-de1be3f65331'}], u'id': u'0dd35572-735a-4ecd-8429-e047ed385100'}
2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver Traceback (most recent call last):
2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver   File "/usr/local/lib/python2.7/dist-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 359, in get_traffic_counters
2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver     chain, wrap=False, zero=True)
2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver   File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/iptables_manager.py", line 669, in get_traffic_counters
2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver     current_table = self.execute(args, run_as_root=True)
2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver   File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in execute
2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver     raise RuntimeError(msg)
2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: iptables: No chain/target/match by that name.
2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver 
2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver 
2016-05-11 18:34:06.858 26409 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'ip', 'netns', 'exec', 'qrouter-0dd35572-735a-4ecd-8429-e047ed385100', 'iptables', '-t', 'filter', '-L', 'neutron-meter-l-d05ce8fb-c01', '-n', '-v', '-x', '-Z'] create_process /usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:82
2016-05-11 18:34:06.911 26409 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: iptables: No chain/target/match by that name.

** Affects: neutron
     Importance: Undecided
     Assignee: HuntXu (huntxu)
         Status: In Progress

** Changed in: neutron
       Status: New => In Progress

** Changed in: neutron
     Assignee: (unassigned) => HuntXu (huntxu)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1580548

Title:
  Metering-agent's routers information is inconsistent with neutron-
  server

Status in neutron:
  In Progress

Bug description:
  With the current implement, under certain curcumstance, router
  information(including internal-stored data, iptables settings) kept
  and set up by metering-agent could be inconsistent with what the
  neutron-server suppose it to be. And that cause the metering-agent to
  run into a problematic status.

  Version:
  Neutron master commit 7327e8c21(Merge "Fix update target tenant RBAC external path")

  Steps to reproduce:
  1. Create internal net, subnet, router. Set external gateway for router, add interface to the router for the created subnet.
  2. Create neutron metering label and rule.
  3. Make sure the created router is added to an active l3-agent.
  4. Create another router and set its external gateway.
  5. Use 'neutron l3-agent-router-remove' to remove the first router from the l3 agent.
  6. Use 'neutron l3-agent-router-add' to add the first router back to the l3 agent by which it was hosted.

  
  (In the following logs, the first created router's id is 0dd35572-735a-4ecd-8429-e047ed385100, the second one is 392820a7-3908-49eb-98a3-967f1638432f)

  Logs in metering-agent.log after step 4:
  2016-05-11 18:32:49.915 26409 DEBUG neutron.agent.linux.utils [req-0515c6f8-cbd1-47d4-a45e-f89e6b6e97b4 - - - - -] Running command: ['sudo', 'ip', 'netns', 'exec', 'qrouter-392820a7-3908-49eb-98a3-967f1638432f', 'iptables-save'] create_process /usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:82
  2016-05-11 18:32:49.926 26409 ERROR neutron.agent.linux.utils [req-0515c6f8-cbd1-47d4-a45e-f89e6b6e97b4 - - - - -] Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-392820a7-3908-49eb-98a3-967f1638432f": No such file or directory

  2016-05-11 18:32:49.926 26409 DEBUG oslo_concurrency.lockutils [req-0515c6f8-cbd1-47d4-a45e-f89e6b6e97b4 - - - - -] Releasing semaphore "iptables-qrouter-392820a7-3908-49eb-98a3-967f1638432f" lock /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:225
  2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent [req-0515c6f8-cbd1-47d4-a45e-f89e6b6e97b4 - - - - -] Driver neutron.services.metering.drivers.iptables.iptables_driver.IptablesMeteringDriver:update_routers runtime error
  2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent Traceback (most recent call last):
  2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent   File "/usr/local/lib/python2.7/dist-packages/neutron/services/metering/agents/metering_agent.py", line 177, in _invoke_driver
  2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent     return getattr(self.metering_driver, func_name)(context, meterings)
  2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent   File "/usr/local/lib/python2.7/dist-packages/oslo_log/helpers.py", line 48, in wrapper
  2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent     return method(*args, **kwargs)
  2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent   File "/usr/local/lib/python2.7/dist-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 125, in update_routers
  2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent     self._process_associate_metering_label(router)
  2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent   File "/usr/local/lib/python2.7/dist-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 228, in _process_associate_metering_label
  2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent     rm.metering_labels[label_id] = label
  2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent   File "/usr/local/lib/python2.7/dist-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 58, in __exit__
  2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent     self.im.apply()
  2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent   File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/iptables_manager.py", line 431, in apply
  2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent     return self._apply()
  2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent   File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/iptables_manager.py", line 439, in _apply
  2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent     return self._apply_synchronized()
  2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent   File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/iptables_manager.py", line 465, in _apply_synchronized
  2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent     save_output = self.execute(args, run_as_root=True)
  2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent   File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in execute
  2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent     raise RuntimeError(msg)
  2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-392820a7-3908-49eb-98a3-967f1638432f": No such file or directory
  2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent
  2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent
  ...Skipping...
  2016-05-11 18:33:06.776 26409 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'ip', 'netns', 'exec', 'qrouter-392820a7-3908-49eb-98a3-967f1638432f', 'iptables', '-t', 'filter', '-L', 'neutron-meter-l-1811c9b9-f9b', '-n', '-v', '-x', '-Z'] create_process /usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:82
  2016-05-11 18:33:06.787 26409 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-392820a7-3908-49eb-98a3-967f1638432f": No such file or directory

  2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver [-] Failed to get traffic counters, router: {u'status': u'ACTIVE', u'name': u'router-two', u'gw_port_id': u'729fba34-4aec-415d-bef8-b93b34e69536', u'admin_state_up': False, u'tenant_id': u'e60fad96858e431cb3da4f0c1ab2d733', u'_metering_labels': [{u'rules': [{u'remote_ip_prefix': u'192.168.88.0/24', u'direction': u'egress', u'metering_label_id': u'1811c9b9-f9b0-40a6-839a-250b20674860', u'id': u'007e3f9e-4e7c-439d-a606-db6ad0ae11ca', u'excluded': False}], u'id': u'1811c9b9-f9b0-40a6-839a-250b20674860'}, {u'rules': [{u'remote_ip_prefix': u'192.168.88.0/24', u'direction': u'ingress', u'metering_label_id': u'd05ce8fb-c013-4ae4-bac5-de1be3f65331', u'id': u'6c580a70-5dec-4225-ade8-44e03fdc9f1b', u'excluded': False}], u'id': u'd05ce8fb-c013-4ae4-bac5-de1be3f65331'}], u'id': u'392820a7-3908-49eb-98a3-967f1638432f'}
  2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver Traceback (most recent call last):
  2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver   File "/usr/local/lib/python2.7/dist-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 359, in get_traffic_counters
  2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver     chain, wrap=False, zero=True)
  2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver   File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/iptables_manager.py", line 669, in get_traffic_counters
  2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver     current_table = self.execute(args, run_as_root=True)
  2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver   File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in execute
  2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver     raise RuntimeError(msg)
  2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-392820a7-3908-49eb-98a3-967f1638432f": No such file or directory
  2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver
  2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver
  2016-05-11 18:33:06.790 26409 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'ip', 'netns', 'exec', 'qrouter-392820a7-3908-49eb-98a3-967f1638432f', 'iptables', '-t', 'filter', '-L', 'neutron-meter-l-d05ce8fb-c01', '-n', '-v', '-x', '-Z'] create_process /usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:82

  
  Logs in metering-agent.log after step 5:
  2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver [-] Failed to get traffic counters, router: {u'status': u'ACTIVE', u'name': u'router-two', u'gw_port_id': u'729fba34-4aec-415d-bef8-b93b34e69536', u'admin_state_up': False, u'tenant_id': u'e60fad96858e431cb3da4f0c1ab2d733', u'_metering_labels': [{u'rules': [{u'remote_ip_prefix': u'192.168.88.0/24', u'direction': u'egress', u'metering_label_id': u'1811c9b9-f9b0-40a6-839a-250b20674860', u'id': u'007e3f9e-4e7c-439d-a606-db6ad0ae11ca', u'excluded': False}], u'id': u'1811c9b9-f9b0-40a6-839a-250b20674860'}, {u'rules': [{u'remote_ip_prefix': u'192.168.88.0/24', u'direction': u'ingress', u'metering_label_id': u'd05ce8fb-c013-4ae4-bac5-de1be3f65331', u'id': u'6c580a70-5dec-4225-ade8-44e03fdc9f1b', u'excluded': False}], u'id': u'd05ce8fb-c013-4ae4-bac5-de1be3f65331'}], u'id': u'392820a7-3908-49eb-98a3-967f1638432f'}
  2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver Traceback (most recent call last):
  2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver   File "/usr/local/lib/python2.7/dist-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 359, in get_traffic_counters
  2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver     chain, wrap=False, zero=True)
  2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver   File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/iptables_manager.py", line 669, in get_traffic_counters
  2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver     current_table = self.execute(args, run_as_root=True)
  2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver   File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in execute
  2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver     raise RuntimeError(msg)
  2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-392820a7-3908-49eb-98a3-967f1638432f": No such file or directory
  2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver 
  2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver 
  2016-05-11 18:33:36.807 26409 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'ip', 'netns', 'exec', 'qrouter-0dd35572-735a-4ecd-8429-e047ed385100', 'iptables', '-t', 'filter', '-L', 'neutron-meter-l-1811c9b9-f9b', '-n', '-v', '-x', '-Z'] create_process /usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:82
  2016-05-11 18:33:36.818 26409 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-0dd35572-735a-4ecd-8429-e047ed385100": No such file or directory

  
  Logs in metering-agent.log after step 6:
  2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver [-] Failed to get traffic counters, router: {u'status': u'ACTIVE', u'name': u'router-one', u'gw_port_id': u'a4137128-27d6-43d6-b04d-a5df8ea62b02', u'admin_state_up': True, u'tenant_id': u'e60fad96858e431cb3da4f0c1ab2d733', u'_metering_labels': [{u'rules': [{u'remote_ip_prefix': u'192.168.88.0/24', u'direction': u'egress', u'metering_label_id': u'1811c9b9-f9b0-40a6-839a-250b20674860', u'id': u'007e3f9e-4e7c-439d-a606-db6ad0ae11ca', u'excluded': False}], u'id': u'1811c9b9-f9b0-40a6-839a-250b20674860'}, {u'rules': [{u'remote_ip_prefix': u'192.168.88.0/24', u'direction': u'ingress', u'metering_label_id': u'd05ce8fb-c013-4ae4-bac5-de1be3f65331', u'id': u'6c580a70-5dec-4225-ade8-44e03fdc9f1b', u'excluded': False}], u'id': u'd05ce8fb-c013-4ae4-bac5-de1be3f65331'}], u'id': u'0dd35572-735a-4ecd-8429-e047ed385100'}
  2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver Traceback (most recent call last):
  2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver   File "/usr/local/lib/python2.7/dist-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 359, in get_traffic_counters
  2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver     chain, wrap=False, zero=True)
  2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver   File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/iptables_manager.py", line 669, in get_traffic_counters
  2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver     current_table = self.execute(args, run_as_root=True)
  2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver   File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in execute
  2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver     raise RuntimeError(msg)
  2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: iptables: No chain/target/match by that name.
  2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver 
  2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver 
  2016-05-11 18:34:06.858 26409 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'ip', 'netns', 'exec', 'qrouter-0dd35572-735a-4ecd-8429-e047ed385100', 'iptables', '-t', 'filter', '-L', 'neutron-meter-l-d05ce8fb-c01', '-n', '-v', '-x', '-Z'] create_process /usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:82
  2016-05-11 18:34:06.911 26409 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: iptables: No chain/target/match by that name.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1580548/+subscriptions
Follow ups

[Bug 1580548] Re: Metering-agent's routers information is inconsistent with neutron-server
From: OpenStack Infra, 2016-11-23