yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1635156] Re: fwaas v2 hit driver internal error when restart l3 agent

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1635156@xxxxxxxxxxxxxxxxxx>
Date: Tue, 08 Nov 2016 13:17:40 -0000
Reply-to: Bug 1635156 <1635156@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.openstack.org/389058
Committed: https://git.openstack.org/cgit/openstack/neutron-fwaas/commit/?id=7de84412592562e1a1b9a0f67caaaad665c9bc9b
Submitter: Jenkins
Branch:    master

commit 7de84412592562e1a1b9a0f67caaaad665c9bc9b
Author: ZhaoBo <zhaobo6@xxxxxxxxxx>
Date:   Thu Oct 20 15:57:07 2016 +0800

    Fix fwaas_v2 driver internal error when restart l3 agent
    
    The fwaas_v2 driver could not process the input 'apply_list'
    just include port_id. The expected input should be like
    [(router_info object, port_id),..]
    
    This patch fixes the l3 agent sync process with server. Make the
    input value meet the required.
    
    Change-Id: I7891b6c65524afd9eec89aec06d1e6bf6b0f0fae
    Closes-Bug: #1635156


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1635156

Title:
  fwaas v2 hit driver internal error when restart l3 agent

Status in neutron:
  Fix Released

Bug description:
  There is a router serviced by l3 agent. And a firewall group located a
  port which serviced by the router, such as an interface of an internal
  subnet. Then I restart l3 agent, it will sync the server side router
  info and refresh the all related router info, include firewall
  iptables rules. It hit driver internal error during this period.

  The trace like:
  2016-10-20 15:05:11.587 DEBUG neutron.agent.linux.iptables_manager [-] IPTablesManager.apply completed with success. 16 iptables commands were issued from (pid=81933) _apply_synchronized /opt/stack/neutron/neutron/agent/linux/iptables_manager.py:533
  2016-10-20 15:05:11.588 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "iptables-qrouter-cc5ab5a3-ef25-4496-87c7-5063cd167ce6" from (pid=81933) lock /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:225
  2016-10-20 15:05:11.589 DEBUG neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2 [-] Process router update, router_id: cc5ab5a3-ef25-4496-87c7-5063cd167ce6  tenant: 488da3aab0ff45df9e85e17e7f89fedd. from (pid=81933) _process_router_update /opt/stack/neutron-fwaas/neutron_fwaas/services/firewall/agents/l3reference/firewall_l3_agent_v2.py:239
  2016-10-20 15:05:11.589 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'list'] from (pid=81933) execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:100
  2016-10-20 15:05:11.593 DEBUG neutron.agent.linux.utils [-] Exit code: 0 from (pid=81933) execute /opt/stack/neutron/neutron/agent/linux/utils.py:141
  2016-10-20 15:05:14.358 DEBUG oslo_policy._cache_handler [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Reloading cached file /etc/neutron/policy.json from (pid=81933) read_cached_file /usr/local/lib/python2.7/dist-packages/oslo_policy/_cache_handler.py:38
  2016-10-20 15:05:14.470 DEBUG oslo_policy.policy [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Reloaded policy file: /etc/neutron/policy.json from (pid=81933) _load_policy_file /usr/local/lib/python2.7/dist-packages/oslo_policy/policy.py:584
  2016-10-20 15:05:14.471 DEBUG oslo_policy._cache_handler [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Reloading cached file /etc/neutron/policy.d/bgpvpn.conf from (pid=81933) read_cached_file /usr/local/lib/python2.7/dist-packages/oslo_policy/_cache_handler.py:38
  2016-10-20 15:05:14.484 DEBUG oslo_policy.policy [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Reloaded policy file: /etc/neutron/policy.d/bgpvpn.conf from (pid=81933) _load_policy_file /usr/local/lib/python2.7/dist-packages/oslo_policy/policy.py:584
  2016-10-20 15:05:14.484 DEBUG oslo_policy._cache_handler [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Reloading cached file /etc/neutron/policy.d/dynamic_routing.conf from (pid=81933) read_cached_file /usr/local/lib/python2.7/dist-packages/oslo_policy/_cache_handler.py:38
  2016-10-20 15:05:14.491 DEBUG oslo_policy.policy [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Reloaded policy file: /etc/neutron/policy.d/dynamic_routing.conf from (pid=81933) _load_policy_file /usr/local/lib/python2.7/dist-packages/oslo_policy/policy.py:584
  2016-10-20 15:05:14.491 DEBUG oslo_policy._cache_handler [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Reloading cached file /etc/neutron/policy.d/neutron-fwaas.json from (pid=81933) read_cached_file /usr/local/lib/python2.7/dist-packages/oslo_policy/_cache_handler.py:38
  2016-10-20 15:05:14.502 DEBUG oslo_policy.policy [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Reloaded policy file: /etc/neutron/policy.d/neutron-fwaas.json from (pid=81933) _load_policy_file /usr/local/lib/python2.7/dist-packages/oslo_policy/policy.py:584
  2016-10-20 15:05:14.503 DEBUG neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2 [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Fetch firewall groups from plugin from (pid=81933) get_firewall_groups_for_project /opt/stack/neutron-fwaas/neutron_fwaas/services/firewall/agents/l3reference/firewall_l3_agent_v2.py:43
  2016-10-20 15:05:14.504 DEBUG oslo_messaging._drivers.amqpdriver [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] CALL msg_id: 2d47fe084441434db22bbc5e21861abc exchange 'neutron' topic 'q-firewall-plugin' from (pid=81933) _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448
  2016-10-20 15:05:14.505 DEBUG neutron.agent.linux.utils [-] Exit code: 0 from (pid=81933) execute /opt/stack/neutron/neutron/agent/linux/utils.py:141
  2016-10-20 15:05:14.505 DEBUG neutron.agent.linux.utils [-] Exit code: 0 from (pid=81933) execute /opt/stack/neutron/neutron/agent/linux/utils.py:141
  2016-10-20 15:05:14.506 DEBUG neutron.agent.linux.utils [-] Exit code: 0 from (pid=81933) execute /opt/stack/neutron/neutron/agent/linux/utils.py:141
  2016-10-20 15:05:14.506 DEBUG neutron.agent.linux.utils [-] Exit code: 0 from (pid=81933) execute /opt/stack/neutron/neutron/agent/linux/utils.py:141
  2016-10-20 15:05:14.507 DEBUG neutron.agent.linux.utils [-] Exit code: 0 from (pid=81933) execute /opt/stack/neutron/neutron/agent/linux/utils.py:141
  2016-10-20 15:05:14.508 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-a1662f95-21a5-494c-b14b-a6249e7179b7', 'ip6tables-save'] from (pid=81933) execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:100
  2016-10-20 15:05:14.525 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 2d47fe084441434db22bbc5e21861abc from (pid=81933) __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:296
  2016-10-20 15:05:17.668 DEBUG neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas_v2 [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Updating firewall 2932b3d9-3a7b-48a1-a16c-bf9f7b2751a5 for tenant 488da3aab0ff45df9e85e17e7f89fedd from (pid=81933) update_firewall_group /opt/stack/neutron-fwaas/neutron_fwaas/services/firewall/drivers/linux/iptables_fwaas_v2.py:131
  2016-10-20 15:05:17.668 ERROR neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas_v2 [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Failed to update firewall: 2932b3d9-3a7b-48a1-a16c-bf9f7b2751a5
  2016-10-20 15:05:17.668 TRACE neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas_v2 Traceback (most recent call last):
  2016-10-20 15:05:17.668 TRACE neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas_v2   File "/opt/stack/neutron-fwaas/neutron_fwaas/services/firewall/drivers/linux/iptables_fwaas_v2.py", line 139, in update_firewall_group
  2016-10-20 15:05:17.668 TRACE neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas_v2     apply_list, firewall)
  2016-10-20 15:05:17.668 TRACE neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas_v2   File "/opt/stack/neutron-fwaas/neutron_fwaas/services/firewall/drivers/linux/iptables_fwaas_v2.py", line 304, in _remove_conntrack_new_firewall
  2016-10-20 15:05:17.668 TRACE neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas_v2     routers_list = list(set([apply_info[0] for apply_info in apply_list]))
  2016-10-20 15:05:17.668 TRACE neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas_v2 KeyError: 0
  2016-10-20 15:05:17.668 TRACE neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas_v2 
  2016-10-20 15:05:17.669 ERROR neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2 [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] FWaaS driver error on ACTIVE for firewall group: 2932b3d9-3a7b-48a1-a16c-bf9f7b2751a5
  2016-10-20 15:05:17.669 TRACE neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2 Traceback (most recent call last):
  2016-10-20 15:05:17.669 TRACE neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2   File "/opt/stack/neutron-fwaas/neutron_fwaas/services/firewall/agents/l3reference/firewall_l3_agent_v2.py", line 192, in _invoke_driver_for_sync_from_plugin
  2016-10-20 15:05:17.669 TRACE neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2     if firewall_group['status'] == n_const.PENDING_DELETE:
  2016-10-20 15:05:17.669 TRACE neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2   File "/opt/stack/neutron-fwaas/neutron_fwaas/services/firewall/drivers/linux/iptables_fwaas_v2.py", line 147, in update_firewall_group
  2016-10-20 15:05:17.669 TRACE neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2     raise fw_ext.FirewallInternalDriverError(driver=FWAAS_DRIVER_NAME)
  2016-10-20 15:05:17.669 TRACE neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2 FirewallInternalDriverError: Fwaas iptables driver: Internal driver error.
  2016-10-20 15:05:17.669 TRACE neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2 
  2016-10-20 15:05:17.670 DEBUG neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2 [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Set firewall groups from plugin from (pid=81933) set_firewall_group_status /opt/stack/neutron-fwaas/neutron_fwaas/services/firewall/agents/l3reference/firewall_l3_agent_v2.py:67
  2016-10-20 15:05:17.671 DEBUG oslo_messaging._drivers.amqpdriver [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] CALL msg_id: f68ec0a4772941ebaacf410f5187c130 exchange 'neutron' topic 'q-firewall-plugin' from (pid=81933) _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1635156/+subscriptions
References

[Bug 1635156] [NEW] fwaas v2 hit driver internal error when restart l3 agent
From: zhaobo, 2016-10-20