yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #58994
[Bug 1580548] Re: Metering-agent's routers information is inconsistent with neutron-server
Reviewed: https://review.openstack.org/315389
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7962dd49eff154acc5d69a6e2a59322a0b2a623b
Submitter: Jenkins
Branch: master
commit 7962dd49eff154acc5d69a6e2a59322a0b2a623b
Author: Hunt Xu <mhuntxu@xxxxxxxxx>
Date: Thu May 12 15:05:24 2016 +0800
Metering: sync only active routers hosted on the same host
When syncing data from neutron server, metering-agent may receive
information about routers that are not hosted by the l3-agent on the
same host, because the server didn't filter them out. This could lead to
the following problems:
* metering-agent tries to setup iptables rules for a router that is not
on the host
* metering-agent tries to get get traffic counters for a router that
was once on the host but is already removed
* metering-agent not sets up iptables rules for a router that is
removed then added back to the host, because nothing about the
router is changed from metering-agent's perspective
This commit fixes the aforementioned problems by making metering-agent
only receive information about routers that are on the same host, and
update metering-agent's knowledge about which routers it should care.
However, there could still be problem if one removes then adds a router
back to the same l3-agent, or just sets the router's admin_state_up
property to False then True in a short time(shorter than the interval
between two syncs). Because the metering-agent sees nothing changed
while during the same time the router's namespace is removed and added
back on the host. Thus metering-agent will fail to get such router's
traffic counters. This commit also make iptables-driver to forget such
routers and leave the metering-agent to reconfigure them later.
Closes-Bug: #1580548
Change-Id: Ia6ae82c676582b06710d6f96b9938c215258182d
Signed-off-by: Hunt Xu <mhuntxu@xxxxxxxxx>
** Changed in: neutron
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1580548
Title:
Metering-agent's routers information is inconsistent with neutron-
server
Status in neutron:
Fix Released
Bug description:
With the current implement, under certain curcumstance, router
information(including internal-stored data, iptables settings) kept
and set up by metering-agent could be inconsistent with what the
neutron-server suppose it to be. And that cause the metering-agent to
run into a problematic status.
Version:
Neutron master commit 7327e8c21(Merge "Fix update target tenant RBAC external path")
Steps to reproduce:
1. Create internal net, subnet, router. Set external gateway for router, add interface to the router for the created subnet.
2. Create neutron metering label and rule.
3. Make sure the created router is added to an active l3-agent.
4. Create another router with --admin_state_up=False, and set its external gateway
5. Use 'neutron l3-agent-router-remove' to remove the first router from the l3 agent.
6. Use 'neutron l3-agent-router-add' to add the first router back to the l3 agent by which it was hosted.
(In the following logs, the first created router's id is 0dd35572
-735a-4ecd-8429-e047ed385100, the second one is 392820a7-3908-49eb-
98a3-967f1638432f)
Logs in metering-agent.log after step 4:
2016-05-11 18:32:49.915 26409 DEBUG neutron.agent.linux.utils [req-0515c6f8-cbd1-47d4-a45e-f89e6b6e97b4 - - - - -] Running command: ['sudo', 'ip', 'netns', 'exec', 'qrouter-392820a7-3908-49eb-98a3-967f1638432f', 'iptables-save'] create_process /usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:82
2016-05-11 18:32:49.926 26409 ERROR neutron.agent.linux.utils [req-0515c6f8-cbd1-47d4-a45e-f89e6b6e97b4 - - - - -] Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-392820a7-3908-49eb-98a3-967f1638432f": No such file or directory
2016-05-11 18:32:49.926 26409 DEBUG oslo_concurrency.lockutils [req-0515c6f8-cbd1-47d4-a45e-f89e6b6e97b4 - - - - -] Releasing semaphore "iptables-qrouter-392820a7-3908-49eb-98a3-967f1638432f" lock /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:225
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent [req-0515c6f8-cbd1-47d4-a45e-f89e6b6e97b4 - - - - -] Driver neutron.services.metering.drivers.iptables.iptables_driver.IptablesMeteringDriver:update_routers runtime error
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent Traceback (most recent call last):
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent File "/usr/local/lib/python2.7/dist-packages/neutron/services/metering/agents/metering_agent.py", line 177, in _invoke_driver
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent return getattr(self.metering_driver, func_name)(context, meterings)
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent File "/usr/local/lib/python2.7/dist-packages/oslo_log/helpers.py", line 48, in wrapper
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent return method(*args, **kwargs)
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent File "/usr/local/lib/python2.7/dist-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 125, in update_routers
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent self._process_associate_metering_label(router)
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent File "/usr/local/lib/python2.7/dist-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 228, in _process_associate_metering_label
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent rm.metering_labels[label_id] = label
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent File "/usr/local/lib/python2.7/dist-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 58, in __exit__
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent self.im.apply()
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/iptables_manager.py", line 431, in apply
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent return self._apply()
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/iptables_manager.py", line 439, in _apply
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent return self._apply_synchronized()
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/iptables_manager.py", line 465, in _apply_synchronized
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent save_output = self.execute(args, run_as_root=True)
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in execute
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent raise RuntimeError(msg)
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-392820a7-3908-49eb-98a3-967f1638432f": No such file or directory
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent
2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent
...Skipping...
2016-05-11 18:33:06.776 26409 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'ip', 'netns', 'exec', 'qrouter-392820a7-3908-49eb-98a3-967f1638432f', 'iptables', '-t', 'filter', '-L', 'neutron-meter-l-1811c9b9-f9b', '-n', '-v', '-x', '-Z'] create_process /usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:82
2016-05-11 18:33:06.787 26409 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-392820a7-3908-49eb-98a3-967f1638432f": No such file or directory
2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver [-] Failed to get traffic counters, router: {u'status': u'ACTIVE', u'name': u'router-two', u'gw_port_id': u'729fba34-4aec-415d-bef8-b93b34e69536', u'admin_state_up': False, u'tenant_id': u'e60fad96858e431cb3da4f0c1ab2d733', u'_metering_labels': [{u'rules': [{u'remote_ip_prefix': u'192.168.88.0/24', u'direction': u'egress', u'metering_label_id': u'1811c9b9-f9b0-40a6-839a-250b20674860', u'id': u'007e3f9e-4e7c-439d-a606-db6ad0ae11ca', u'excluded': False}], u'id': u'1811c9b9-f9b0-40a6-839a-250b20674860'}, {u'rules': [{u'remote_ip_prefix': u'192.168.88.0/24', u'direction': u'ingress', u'metering_label_id': u'd05ce8fb-c013-4ae4-bac5-de1be3f65331', u'id': u'6c580a70-5dec-4225-ade8-44e03fdc9f1b', u'excluded': False}], u'id': u'd05ce8fb-c013-4ae4-bac5-de1be3f65331'}], u'id': u'392820a7-3908-49eb-98a3-967f1638432f'}
2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver Traceback (most recent call last):
2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver File "/usr/local/lib/python2.7/dist-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 359, in get_traffic_counters
2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver chain, wrap=False, zero=True)
2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/iptables_manager.py", line 669, in get_traffic_counters
2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver current_table = self.execute(args, run_as_root=True)
2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in execute
2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver raise RuntimeError(msg)
2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-392820a7-3908-49eb-98a3-967f1638432f": No such file or directory
2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver
2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver
2016-05-11 18:33:06.790 26409 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'ip', 'netns', 'exec', 'qrouter-392820a7-3908-49eb-98a3-967f1638432f', 'iptables', '-t', 'filter', '-L', 'neutron-meter-l-d05ce8fb-c01', '-n', '-v', '-x', '-Z'] create_process /usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:82
Logs in metering-agent.log after step 5:
2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver [-] Failed to get traffic counters, router: {u'status': u'ACTIVE', u'name': u'router-two', u'gw_port_id': u'729fba34-4aec-415d-bef8-b93b34e69536', u'admin_state_up': False, u'tenant_id': u'e60fad96858e431cb3da4f0c1ab2d733', u'_metering_labels': [{u'rules': [{u'remote_ip_prefix': u'192.168.88.0/24', u'direction': u'egress', u'metering_label_id': u'1811c9b9-f9b0-40a6-839a-250b20674860', u'id': u'007e3f9e-4e7c-439d-a606-db6ad0ae11ca', u'excluded': False}], u'id': u'1811c9b9-f9b0-40a6-839a-250b20674860'}, {u'rules': [{u'remote_ip_prefix': u'192.168.88.0/24', u'direction': u'ingress', u'metering_label_id': u'd05ce8fb-c013-4ae4-bac5-de1be3f65331', u'id': u'6c580a70-5dec-4225-ade8-44e03fdc9f1b', u'excluded': False}], u'id': u'd05ce8fb-c013-4ae4-bac5-de1be3f65331'}], u'id': u'392820a7-3908-49eb-98a3-967f1638432f'}
2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver Traceback (most recent call last):
2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver File "/usr/local/lib/python2.7/dist-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 359, in get_traffic_counters
2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver chain, wrap=False, zero=True)
2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/iptables_manager.py", line 669, in get_traffic_counters
2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver current_table = self.execute(args, run_as_root=True)
2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in execute
2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver raise RuntimeError(msg)
2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-392820a7-3908-49eb-98a3-967f1638432f": No such file or directory
2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver
2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver
2016-05-11 18:33:36.807 26409 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'ip', 'netns', 'exec', 'qrouter-0dd35572-735a-4ecd-8429-e047ed385100', 'iptables', '-t', 'filter', '-L', 'neutron-meter-l-1811c9b9-f9b', '-n', '-v', '-x', '-Z'] create_process /usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:82
2016-05-11 18:33:36.818 26409 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-0dd35572-735a-4ecd-8429-e047ed385100": No such file or directory
Logs in metering-agent.log after step 6:
2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver [-] Failed to get traffic counters, router: {u'status': u'ACTIVE', u'name': u'router-one', u'gw_port_id': u'a4137128-27d6-43d6-b04d-a5df8ea62b02', u'admin_state_up': True, u'tenant_id': u'e60fad96858e431cb3da4f0c1ab2d733', u'_metering_labels': [{u'rules': [{u'remote_ip_prefix': u'192.168.88.0/24', u'direction': u'egress', u'metering_label_id': u'1811c9b9-f9b0-40a6-839a-250b20674860', u'id': u'007e3f9e-4e7c-439d-a606-db6ad0ae11ca', u'excluded': False}], u'id': u'1811c9b9-f9b0-40a6-839a-250b20674860'}, {u'rules': [{u'remote_ip_prefix': u'192.168.88.0/24', u'direction': u'ingress', u'metering_label_id': u'd05ce8fb-c013-4ae4-bac5-de1be3f65331', u'id': u'6c580a70-5dec-4225-ade8-44e03fdc9f1b', u'excluded': False}], u'id': u'd05ce8fb-c013-4ae4-bac5-de1be3f65331'}], u'id': u'0dd35572-735a-4ecd-8429-e047ed385100'}
2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver Traceback (most recent call last):
2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver File "/usr/local/lib/python2.7/dist-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 359, in get_traffic_counters
2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver chain, wrap=False, zero=True)
2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/iptables_manager.py", line 669, in get_traffic_counters
2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver current_table = self.execute(args, run_as_root=True)
2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in execute
2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver raise RuntimeError(msg)
2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: iptables: No chain/target/match by that name.
2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver
2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver
2016-05-11 18:34:06.858 26409 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'ip', 'netns', 'exec', 'qrouter-0dd35572-735a-4ecd-8429-e047ed385100', 'iptables', '-t', 'filter', '-L', 'neutron-meter-l-d05ce8fb-c01', '-n', '-v', '-x', '-Z'] create_process /usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:82
2016-05-11 18:34:06.911 26409 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: iptables: No chain/target/match by that name.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1580548/+subscriptions
References
