yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1811405] Re: OVSFWTagNotFound raised many times in neutron-openvswitch-agent logs

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1811405@xxxxxxxxxxxxxxxxxx>
Date: Mon, 11 Mar 2019 12:41:35 -0000
Reply-to: Bug 1811405 <1811405@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.openstack.org/630910
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=22f55822aab17c758f5f723c304b1acaf3179ec1
Submitter: Zuul
Branch:    master

commit 22f55822aab17c758f5f723c304b1acaf3179ec1
Author: Lajos Katona <lajos.katona@xxxxxxxxxxxx>
Date:   Tue Jan 15 09:20:54 2019 +0100

    Catch OVSFWTagNotFound in update_port_filter
    
    In neutron.agent.linux.openvswitch_firewall.firewall make the method
    update_port_filter catch OVSFWTagNotFound and log it to avoid
    traceback in log files.
    
    Change-Id: I584d867f0e1c47447cb8790fd715fa01ec902438
    Closes-Bug: #1811405


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1811405

Title:
  OVSFWTagNotFound raised many times in neutron-openvswitch-agent logs

Status in neutron:
  Fix Released

Bug description:
  I found out that recently exception like:

  Jan 11 08:56:11.074296 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-207e9202-dfc1-4b0d-b9d4-85eaec6174ee None None] Error while processing VIF ports: neutron.agent.linux.openvswitch_firewall.exceptions.OVSFWTagNotFound: Cannot get tag for port tap20acc597-23 from its other_config: {}
  Jan 11 08:56:11.074665 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  Jan 11 08:56:11.074999 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 526, in get_or_create_ofport
  Jan 11 08:56:11.075349 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     of_port = self.sg_port_map.ports[port_id]
  Jan 11 08:56:11.075713 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError: '20acc597-23f0-4c18-980e-061aa6494ccf'
  Jan 11 08:56:11.076036 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
  Jan 11 08:56:11.077854 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling of the above exception, another exception occurred:
  Jan 11 08:56:11.078269 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
  Jan 11 08:56:11.078611 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  Jan 11 08:56:11.079337 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 79, in get_tag_from_other_config
  Jan 11 08:56:11.079670 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return int(other_config['tag'])
  Jan 11 08:56:11.080000 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError: 'tag'
  Jan 11 08:56:11.080351 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
  Jan 11 08:56:11.080677 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling of the above exception, another exception occurred:
  Jan 11 08:56:11.081027 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
  Jan 11 08:56:11.081354 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  Jan 11 08:56:11.081681 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2196, in rpc_loop
  Jan 11 08:56:11.082077 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info, provisioning_needed)
  Jan 11 08:56:11.082475 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/local/lib/python3.5/dist-packages/osprofiler/profiler.py", line 160, in wrapper
  Jan 11 08:56:11.082907 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = f(*args, **kwargs)
  Jan 11 08:56:11.083233 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1770, in process_network_ports
  Jan 11 08:56:11.083560 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info.get('updated', set()))
  Jan 11 08:56:11.083903 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 268, in setup_port_filters
  Jan 11 08:56:11.084232 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.refresh_firewall(updated_devices)
  Jan 11 08:56:11.084563 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 122, in decorated_function
  Jan 11 08:56:11.084901 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     *args, **kwargs)
  Jan 11 08:56:11.085230 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 221, in refresh_firewall
  Jan 11 08:56:11.085575 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self._apply_port_filter(device_ids, update_filter=True)
  Jan 11 08:56:11.085961 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 153, in _apply_port_filter
  Jan 11 08:56:11.086287 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.firewall.update_port_filter(device)
  Jan 11 08:56:11.086680 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 581, in update_port_filter
  Jan 11 08:56:11.086999 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     of_port = self.get_or_create_ofport(port)
  Jan 11 08:56:11.087321 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 528, in get_or_create_ofport
  Jan 11 08:56:11.087936 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_vlan_id = self._get_port_vlan_tag(ovs_port.port_name)
  Jan 11 08:56:11.088270 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 512, in _get_port_vlan_tag
  Jan 11 08:56:11.089768 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return get_tag_from_other_config(self.int_br.br, port_name)
  Jan 11 08:56:11.090081 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 82, in get_tag_from_other_config
  Jan 11 08:56:11.090328 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_name=port_name, other_config=other_config)
  Jan 11 08:56:11.090619 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent neutron.agent.linux.openvswitch_firewall.exceptions.OVSFWTagNotFound: Cannot get tag for port tap20acc597-23 from its other_config: {}

  Is appearing in q-agt logs in CI jobs quite often.
  Example: http://logs.openstack.org/41/626641/17/check/grenade-py3/50d0a7b/logs/screen-q-agt.txt.gz?level=ERROR

  I'm not sure if this is direct reason of test failures but we should
  avoid such exceptions in logs IMHO if it is "normal" case.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1811405/+subscriptions
References

[Bug 1811405] [NEW] OVSFWTagNotFound raised many times in neutron-openvswitch-agent logs
From: Slawek Kaplonski, 2019-01-11