yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1740885] [NEW] Trunk ports are sometimes not tagged with internal vlan

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Jakub Libosvar <libosvar@xxxxxxxxxx>
Date: Tue, 02 Jan 2018 15:50:06 -0000
Reply-to: Bug 1740885 <1740885@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

It happens that tpi patch ports between trunk bridge and integration
bridge don't have internal vlan tag in other_config row in ovsdb. It
looks like a race between trunk handler and ovs agent.

Example of failure: http://logs.openstack.org/92/527992/2/check/neutron-
tempest-plugin-dvr-multinode-
scenario/166eee3/logs/subnode-2/screen-q-agt.txt.gz#_Dec_14_18_31_25_801432

Trace example: 
Dec 14 18:31:25.801432 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-412981e2-ac8a-4fe8-8ec4-288bbb63e2a7 None None] Error while processing VIF ports: OVSFWTagNotFound: Cannot get tag for port tpi-6457d45d-b6 from its other_config: {}
Dec 14 18:31:25.801580 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
Dec 14 18:31:25.801708 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2080, in rpc_loop
Dec 14 18:31:25.801838 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info, ovs_restarted)
Dec 14 18:31:25.801965 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 157, in wrapper
Dec 14 18:31:25.802089 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = f(*args, **kwargs)
Dec 14 18:31:25.802214 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1676, in process_network_ports
Dec 14 18:31:25.802345 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info.get('updated', set()))
Dec 14 18:31:25.802476 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 256, in setup_port_filters
Dec 14 18:31:25.802600 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.refresh_firewall(updated_devices)
Dec 14 18:31:25.802725 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 110, in decorated_function
Dec 14 18:31:25.802850 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     *args, **kwargs)
Dec 14 18:31:25.802983 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 209, in refresh_firewall
Dec 14 18:31:25.803103 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self._apply_port_filter(device_ids, update_filter=True)
Dec 14 18:31:25.803237 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 141, in _apply_port_filter
Dec 14 18:31:25.803366 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.firewall.update_port_filter(device)
Dec 14 18:31:25.803492 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 509, in update_port_filter
Dec 14 18:31:25.803612 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.prepare_port_filter(port)
Dec 14 18:31:25.803763 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 492, in prepare_port_filter
Dec 14 18:31:25.803894 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     of_port = self.get_or_create_ofport(port)
Dec 14 18:31:25.804021 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 464, in get_or_create_ofport
Dec 14 18:31:25.804152 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_vlan_id = self._get_port_vlan_tag(ovs_port.port_name)
Dec 14 18:31:25.804275 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 448, in _get_port_vlan_tag
Dec 14 18:31:25.804414 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return get_tag_from_other_config(self.int_br.br, port_name)
Dec 14 18:31:25.804548 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 74, in get_tag_from_other_config
Dec 14 18:31:25.804670 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_name=port_name, other_config=other_config)
Dec 14 18:31:25.804794 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent OVSFWTagNotFound: Cannot get tag for port tpi-6457d45d-b6 from its other_config: {}
Dec 14 18:31:25.804923 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent

** Affects: neutron
     Importance: High
     Assignee: Jakub Libosvar (libosvar)
         Status: Confirmed


** Tags: ovs-fw sg-fw trunk

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1740885

Title:
  Trunk ports are sometimes not tagged with internal vlan

Status in neutron:
  Confirmed

Bug description:
  It happens that tpi patch ports between trunk bridge and integration
  bridge don't have internal vlan tag in other_config row in ovsdb. It
  looks like a race between trunk handler and ovs agent.

  Example of failure: http://logs.openstack.org/92/527992/2/check
  /neutron-tempest-plugin-dvr-multinode-
  scenario/166eee3/logs/subnode-2/screen-q-agt.txt.gz#_Dec_14_18_31_25_801432

  Trace example: 
  Dec 14 18:31:25.801432 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-412981e2-ac8a-4fe8-8ec4-288bbb63e2a7 None None] Error while processing VIF ports: OVSFWTagNotFound: Cannot get tag for port tpi-6457d45d-b6 from its other_config: {}
  Dec 14 18:31:25.801580 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  Dec 14 18:31:25.801708 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2080, in rpc_loop
  Dec 14 18:31:25.801838 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info, ovs_restarted)
  Dec 14 18:31:25.801965 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 157, in wrapper
  Dec 14 18:31:25.802089 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = f(*args, **kwargs)
  Dec 14 18:31:25.802214 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1676, in process_network_ports
  Dec 14 18:31:25.802345 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info.get('updated', set()))
  Dec 14 18:31:25.802476 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 256, in setup_port_filters
  Dec 14 18:31:25.802600 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.refresh_firewall(updated_devices)
  Dec 14 18:31:25.802725 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 110, in decorated_function
  Dec 14 18:31:25.802850 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     *args, **kwargs)
  Dec 14 18:31:25.802983 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 209, in refresh_firewall
  Dec 14 18:31:25.803103 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self._apply_port_filter(device_ids, update_filter=True)
  Dec 14 18:31:25.803237 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 141, in _apply_port_filter
  Dec 14 18:31:25.803366 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.firewall.update_port_filter(device)
  Dec 14 18:31:25.803492 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 509, in update_port_filter
  Dec 14 18:31:25.803612 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.prepare_port_filter(port)
  Dec 14 18:31:25.803763 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 492, in prepare_port_filter
  Dec 14 18:31:25.803894 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     of_port = self.get_or_create_ofport(port)
  Dec 14 18:31:25.804021 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 464, in get_or_create_ofport
  Dec 14 18:31:25.804152 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_vlan_id = self._get_port_vlan_tag(ovs_port.port_name)
  Dec 14 18:31:25.804275 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 448, in _get_port_vlan_tag
  Dec 14 18:31:25.804414 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return get_tag_from_other_config(self.int_br.br, port_name)
  Dec 14 18:31:25.804548 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 74, in get_tag_from_other_config
  Dec 14 18:31:25.804670 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_name=port_name, other_config=other_config)
  Dec 14 18:31:25.804794 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent OVSFWTagNotFound: Cannot get tag for port tpi-6457d45d-b6 from its other_config: {}
  Dec 14 18:31:25.804923 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1740885/+subscriptions
Follow ups

[Bug 1740885] Re: Security group updates fail when port hasn't been initialized yet
From: OpenStack Infra, 2018-01-31