yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1849098] [NEW] ovs agent is stuck with OVSFWTagNotFound when dealing with unbound port

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Oleg Bondarev <obondarev@xxxxxxxxxxxx>
Date: Mon, 21 Oct 2019 11:33:23 -0000
Reply-to: Bug 1849098 <1849098@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

neutron-openvswitch-agent meets unbound port:

2019-10-17 11:32:21.868 135 WARNING
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-
aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Device
ef34215f-e099-4fd0-935f-c9a42951d166 not defined on plugin or binding
failed

Later when applying firewall rules:

2019-10-17 11:32:21.901 135 INFO neutron.agent.securitygroups_rpc [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Preparing filters for devices {'ef34215f-e099-4fd0-935f-c9a42951d166', 'e9c97cf0-1a5e-4d77-b57b-0ba474d12e29', 'fff1bb24-6423-4486-87c4-1fe17c552cca', '2e20f9ee-bcb5-445c-b31f-d70d276d45c9', '03a60047-cb07-42a4-8b49-619d5982a9bd', 'a452cea2-deaf-4411-bbae-ce83870cbad4', '79b03e5c-9be0-4808-9784-cb4878c3dbd5', '9b971e75-3c1b-463d-88cf-3f298105fa6e'}
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Error while processing VIF ports: neutron.agent.linux.openvswitch_firewall.exceptions.OVSFWTagNotFound: Cannot get tag for port o-hm0 from its other_config: {}
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 530, in get_or_create_ofport
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     of_port = self.sg_port_map.ports[port_id]
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError: 'ef34215f-e099-4fd0-935f-c9a42951d166'
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling of the above exception, another exception occurred:
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 81, in get_tag_from_other_config
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return int(other_config['tag'])
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError: 'tag'
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling of the above exception, another exception occurred:
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2280, in rpc_loop
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info, provisioning_needed)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = f(*args, **kwargs)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1847, in process_network_ports
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info.get('updated', set()))
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 258, in setup_port_filters
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.prepare_devices_filter(new_devices)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 125, in decorated_function
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     *args, **kwargs)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 133, in prepare_devices_filter
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self._apply_port_filter(device_ids)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 164, in _apply_port_filter
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.firewall.prepare_port_filter(device)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 555, in prepare_port_filter
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     of_port = self.get_or_create_ofport(port)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 532, in get_or_create_ofport
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_vlan_id = self._get_port_vlan_tag(ovs_port.port_name)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 516, in _get_port_vlan_tag
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return get_tag_from_other_config(self.int_br.br, port_name)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 84, in get_tag_from_other_config
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_name=port_name, other_config=other_config)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent neutron.agent.linux.openvswitch_firewall.exceptions.OVSFWTagNotFound: Cannot get tag for port o-hm0 from its other_config: {}
2019-10-17 11:32:21.909 135 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Agent out of sync with plugin!

this happens in each agent cycle so agent can't do anything.

Need to handle OVSFWTagNotFound in prepare_port_filter() like was done
for update_port_filter in https://review.opendev.org/#/c/630910/

** Affects: neutron
     Importance: High
     Assignee: Oleg Bondarev (obondarev)
         Status: In Progress

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1849098

Title:
  ovs agent is stuck with OVSFWTagNotFound when dealing with unbound
  port

Status in neutron:
  In Progress

Bug description:
  neutron-openvswitch-agent meets unbound port:

  2019-10-17 11:32:21.868 135 WARNING
  neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-
  aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Device
  ef34215f-e099-4fd0-935f-c9a42951d166 not defined on plugin or binding
  failed

  Later when applying firewall rules:

  2019-10-17 11:32:21.901 135 INFO neutron.agent.securitygroups_rpc [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Preparing filters for devices {'ef34215f-e099-4fd0-935f-c9a42951d166', 'e9c97cf0-1a5e-4d77-b57b-0ba474d12e29', 'fff1bb24-6423-4486-87c4-1fe17c552cca', '2e20f9ee-bcb5-445c-b31f-d70d276d45c9', '03a60047-cb07-42a4-8b49-619d5982a9bd', 'a452cea2-deaf-4411-bbae-ce83870cbad4', '79b03e5c-9be0-4808-9784-cb4878c3dbd5', '9b971e75-3c1b-463d-88cf-3f298105fa6e'}
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Error while processing VIF ports: neutron.agent.linux.openvswitch_firewall.exceptions.OVSFWTagNotFound: Cannot get tag for port o-hm0 from its other_config: {}
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 530, in get_or_create_ofport
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     of_port = self.sg_port_map.ports[port_id]
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError: 'ef34215f-e099-4fd0-935f-c9a42951d166'
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling of the above exception, another exception occurred:
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 81, in get_tag_from_other_config
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return int(other_config['tag'])
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError: 'tag'
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling of the above exception, another exception occurred:
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2280, in rpc_loop
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info, provisioning_needed)
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = f(*args, **kwargs)
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1847, in process_network_ports
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info.get('updated', set()))
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 258, in setup_port_filters
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.prepare_devices_filter(new_devices)
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 125, in decorated_function
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     *args, **kwargs)
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 133, in prepare_devices_filter
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self._apply_port_filter(device_ids)
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 164, in _apply_port_filter
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.firewall.prepare_port_filter(device)
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 555, in prepare_port_filter
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     of_port = self.get_or_create_ofport(port)
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 532, in get_or_create_ofport
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_vlan_id = self._get_port_vlan_tag(ovs_port.port_name)
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 516, in _get_port_vlan_tag
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return get_tag_from_other_config(self.int_br.br, port_name)
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 84, in get_tag_from_other_config
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_name=port_name, other_config=other_config)
  2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent neutron.agent.linux.openvswitch_firewall.exceptions.OVSFWTagNotFound: Cannot get tag for port o-hm0 from its other_config: {}
  2019-10-17 11:32:21.909 135 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Agent out of sync with plugin!

  this happens in each agent cycle so agent can't do anything.

  Need to handle OVSFWTagNotFound in prepare_port_filter() like was done
  for update_port_filter in https://review.opendev.org/#/c/630910/

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1849098/+subscriptions
Follow ups

[Bug 1849098] Re: ovs agent is stuck with OVSFWTagNotFound when dealing with unbound port
From: Hemanth Nakkina, 2021-05-21
[Bug 1849098] Re: ovs agent is stuck with OVSFWTagNotFound when dealing with unbound port
From: Corey Bryant, 2021-05-13
[Bug 1849098] Re: ovs agent is stuck with OVSFWTagNotFound when dealing with unbound port
From: Launchpad Bug Tracker, 2021-05-13
[Bug 1849098] Re: ovs agent is stuck with OVSFWTagNotFound when dealing with unbound port
From: Edward Hope-Morley, 2021-05-11
[Bug 1849098] Re: ovs agent is stuck with OVSFWTagNotFound when dealing with unbound port
From: Seyeong Kim, 2021-05-03
[Bug 1849098] Re: ovs agent is stuck with OVSFWTagNotFound when dealing with unbound port
From: OpenStack Infra, 2019-10-27