yahoo-eng-team team mailing list archive

[Jakub Libosvar <libosvar@xxxxxxxxxx>]

Public bug reported:

When the veth pair for metadata namespace is created, OVSDB emits
multiple Port Binding events. The agent re-acts on two and attempts to
spawn haproxy process twice in a small timeframe.

This is an example of port 26d98ec8-ce82-45d7-8043-f00f42b1f15c and
datapath 1d59fe65-eca7-4341-95a1-8fd2dad007ea

2020-03-16 19:50:13.052 29393 DEBUG ovsdbapp.backend.ovs_idl.event [-] Matched UPDATE: PortBindingChassisEvent(events=('update',), table='Port_Binding', conditions=None, old_conditions=None) to row=Port_Binding(tunnel_key=3, parent_port=[], logical_port=26d98ec8-ce82-45d7-8043-f00f42b1f15c, mac=['fa:16:3e:07:36:c9 10.100.0.7'], chassis=[<ovs.db.idl.Row object at 0x7f18124a38d0>], ha_chassis_group=[], options={'requested-chassis': 'compute-0.redhat.local'}, external_ids={'neutron:cidrs': '10.100.0.7/28', 'neutron:device_id': '27584016-6cf4-4d2a-9300-098fb9dc6686', 'neutron:device_owner': 'compute:nova', 'neutron:network_name': 'neutron-c4c60426-9c40-4746-8193-2354be1fa436', 'neutron:port_name': '', 'neutron:project_id': '27c427e573c34229a2651f14da1bf15f', 'neutron:revision_number': '2', 'neutron:security_group_ids': 'c289479c-ee0e-4e52-a475-ec51d04d67d8'}, encap=[], gateway_chassis=[], type=, nat_addresses=[], virtual_parent=[], datapath=1d59fe65-eca7-4341-95a1-8fd2dad007ea, tag=[]) old=Port_Binding(chassis=[]) matches /usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/event.py:44
2020-03-16 19:50:13.053 29393 INFO networking_ovn.agent.metadata.agent [-] Port 26d98ec8-ce82-45d7-8043-f00f42b1f15c in datapath 1d59fe65-eca7-4341-95a1-8fd2dad007ea bound to our chassis
2020-03-16 19:50:13.054 29393 DEBUG networking_ovn.agent.metadata.agent [-] Provisioning datapath 1d59fe65-eca7-4341-95a1-8fd2dad007ea provision_datapath /usr/lib/python3.6/site-packages/networking_ovn/agent/metadata/agent.py:318

2020-03-16 19:50:13.744 29393 DEBUG neutron.agent.linux.utils [-]
Running command: ['sudo', 'neutron-rootwrap',
'/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'ovnmeta-
1d59fe65-eca7-4341-95a1-8fd2dad007ea', 'haproxy', '-f',
'/var/lib/neutron/ovn-metadata-
proxy/1d59fe65-eca7-4341-95a1-8fd2dad007ea.conf'] create_process
/usr/lib/python3.6/site-packages/neutron/agent/linux/utils.py:87

2020-03-16 19:50:14.501 29393 DEBUG ovsdbapp.backend.ovs_idl.event [-] Matched UPDATE: PortBindingChassisEvent(events=('update',), table='Port_Binding', conditions=None, old_conditions=None) to row=Port_Binding(tunnel_key=3, parent_port=[], logical_port=26d98ec8-ce82-45d7-8043-f00f42b1f15c, mac=['fa:16:3e:07:36:c9 10.100.0.7'], chassis=[<ovs.db.idl.Row object at 0x7f18124a38d0>], ha_chassis_group=[], options={'requested-chassis': 'compute-0.redhat.local'}, external_ids={'neutron:cidrs': '10.100.0.7/28', 'neutron:device_id': '27584016-6cf4-4d2a-9300-098fb9dc6686', 'neutron:device_owner': 'compute:nova', 'neutron:network_name': 'neutron-c4c60426-9c40-4746-8193-2354be1fa436', 'neutron:port_name': '', 'neutron:project_id': '27c427e573c34229a2651f14da1bf15f', 'neutron:revision_number': '4', 'neutron:security_group_ids': 'c289479c-ee0e-4e52-a475-ec51d04d67d8'}, encap=[], gateway_chassis=[], type=, nat_addresses=[], virtual_parent=[], datapath=1d59fe65-eca7-4341-95a1-8fd2dad007ea, tag=[]) old=Port_Binding(external_ids={'neutron:cidrs': '10.100.0.7/28', 'neutron:device_id': '27584016-6cf4-4d2a-9300-098fb9dc6686', 'neutron:device_owner': 'compute:nova', 'neutron:network_name': 'neutron-c4c60426-9c40-4746-8193-2354be1fa436', 'neutron:port_name': '', 'neutron:project_id': '27c427e573c34229a2651f14da1bf15f', 'neutron:revision_number': '2', 'neutron:security_group_ids': 'c289479c-ee0e-4e52-a475-ec51d04d67d8'}) matches /usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/event.py:44
2020-03-16 19:50:14.503 29393 INFO networking_ovn.agent.metadata.agent [-] Port 26d98ec8-ce82-45d7-8043-f00f42b1f15c in datapath 1d59fe65-eca7-4341-95a1-8fd2dad007ea bound to our chassis
2020-03-16 19:50:14.505 29393 DEBUG networking_ovn.agent.metadata.agent [-] Provisioning datapath 1d59fe65-eca7-4341-95a1-8fd2dad007ea provision_datapath /usr/lib/python3.6/site-packages/networking_ovn/agent/metadata/agent.py:318
2020-03-16 19:50:14.653 29393 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'ovnmeta-1d59fe65-eca7-4341-95a1-8fd2dad007ea', 'haproxy', '-f', '/var/lib/neutron/ovn-metadata-proxy/1d59fe65-eca7-4341-95a1-8fd2dad007ea.conf'] create_process /usr/lib/python3.6/site-packages/neutron/agent/linux/utils.py:87

** Affects: neutron
     Importance: Undecided
     Assignee: Jakub Libosvar (libosvar)
         Status: New


** Tags: ovn

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1868125

Title:
  [ovn] Metadata agent spawns haproxy quickly twice on a single new port
  binding

Status in neutron:
  New

Bug description:
  When the veth pair for metadata namespace is created, OVSDB emits
  multiple Port Binding events. The agent re-acts on two and attempts to
  spawn haproxy process twice in a small timeframe.

  This is an example of port 26d98ec8-ce82-45d7-8043-f00f42b1f15c and
  datapath 1d59fe65-eca7-4341-95a1-8fd2dad007ea

  2020-03-16 19:50:13.052 29393 DEBUG ovsdbapp.backend.ovs_idl.event [-] Matched UPDATE: PortBindingChassisEvent(events=('update',), table='Port_Binding', conditions=None, old_conditions=None) to row=Port_Binding(tunnel_key=3, parent_port=[], logical_port=26d98ec8-ce82-45d7-8043-f00f42b1f15c, mac=['fa:16:3e:07:36:c9 10.100.0.7'], chassis=[<ovs.db.idl.Row object at 0x7f18124a38d0>], ha_chassis_group=[], options={'requested-chassis': 'compute-0.redhat.local'}, external_ids={'neutron:cidrs': '10.100.0.7/28', 'neutron:device_id': '27584016-6cf4-4d2a-9300-098fb9dc6686', 'neutron:device_owner': 'compute:nova', 'neutron:network_name': 'neutron-c4c60426-9c40-4746-8193-2354be1fa436', 'neutron:port_name': '', 'neutron:project_id': '27c427e573c34229a2651f14da1bf15f', 'neutron:revision_number': '2', 'neutron:security_group_ids': 'c289479c-ee0e-4e52-a475-ec51d04d67d8'}, encap=[], gateway_chassis=[], type=, nat_addresses=[], virtual_parent=[], datapath=1d59fe65-eca7-4341-95a1-8fd2dad007ea, tag=[]) old=Port_Binding(chassis=[]) matches /usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/event.py:44
  2020-03-16 19:50:13.053 29393 INFO networking_ovn.agent.metadata.agent [-] Port 26d98ec8-ce82-45d7-8043-f00f42b1f15c in datapath 1d59fe65-eca7-4341-95a1-8fd2dad007ea bound to our chassis
  2020-03-16 19:50:13.054 29393 DEBUG networking_ovn.agent.metadata.agent [-] Provisioning datapath 1d59fe65-eca7-4341-95a1-8fd2dad007ea provision_datapath /usr/lib/python3.6/site-packages/networking_ovn/agent/metadata/agent.py:318

  2020-03-16 19:50:13.744 29393 DEBUG neutron.agent.linux.utils [-]
  Running command: ['sudo', 'neutron-rootwrap',
  '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'ovnmeta-
  1d59fe65-eca7-4341-95a1-8fd2dad007ea', 'haproxy', '-f',
  '/var/lib/neutron/ovn-metadata-
  proxy/1d59fe65-eca7-4341-95a1-8fd2dad007ea.conf'] create_process
  /usr/lib/python3.6/site-packages/neutron/agent/linux/utils.py:87

  2020-03-16 19:50:14.501 29393 DEBUG ovsdbapp.backend.ovs_idl.event [-] Matched UPDATE: PortBindingChassisEvent(events=('update',), table='Port_Binding', conditions=None, old_conditions=None) to row=Port_Binding(tunnel_key=3, parent_port=[], logical_port=26d98ec8-ce82-45d7-8043-f00f42b1f15c, mac=['fa:16:3e:07:36:c9 10.100.0.7'], chassis=[<ovs.db.idl.Row object at 0x7f18124a38d0>], ha_chassis_group=[], options={'requested-chassis': 'compute-0.redhat.local'}, external_ids={'neutron:cidrs': '10.100.0.7/28', 'neutron:device_id': '27584016-6cf4-4d2a-9300-098fb9dc6686', 'neutron:device_owner': 'compute:nova', 'neutron:network_name': 'neutron-c4c60426-9c40-4746-8193-2354be1fa436', 'neutron:port_name': '', 'neutron:project_id': '27c427e573c34229a2651f14da1bf15f', 'neutron:revision_number': '4', 'neutron:security_group_ids': 'c289479c-ee0e-4e52-a475-ec51d04d67d8'}, encap=[], gateway_chassis=[], type=, nat_addresses=[], virtual_parent=[], datapath=1d59fe65-eca7-4341-95a1-8fd2dad007ea, tag=[]) old=Port_Binding(external_ids={'neutron:cidrs': '10.100.0.7/28', 'neutron:device_id': '27584016-6cf4-4d2a-9300-098fb9dc6686', 'neutron:device_owner': 'compute:nova', 'neutron:network_name': 'neutron-c4c60426-9c40-4746-8193-2354be1fa436', 'neutron:port_name': '', 'neutron:project_id': '27c427e573c34229a2651f14da1bf15f', 'neutron:revision_number': '2', 'neutron:security_group_ids': 'c289479c-ee0e-4e52-a475-ec51d04d67d8'}) matches /usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/event.py:44
  2020-03-16 19:50:14.503 29393 INFO networking_ovn.agent.metadata.agent [-] Port 26d98ec8-ce82-45d7-8043-f00f42b1f15c in datapath 1d59fe65-eca7-4341-95a1-8fd2dad007ea bound to our chassis
  2020-03-16 19:50:14.505 29393 DEBUG networking_ovn.agent.metadata.agent [-] Provisioning datapath 1d59fe65-eca7-4341-95a1-8fd2dad007ea provision_datapath /usr/lib/python3.6/site-packages/networking_ovn/agent/metadata/agent.py:318
  2020-03-16 19:50:14.653 29393 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'ovnmeta-1d59fe65-eca7-4341-95a1-8fd2dad007ea', 'haproxy', '-f', '/var/lib/neutron/ovn-metadata-proxy/1d59fe65-eca7-4341-95a1-8fd2dad007ea.conf'] create_process /usr/lib/python3.6/site-packages/neutron/agent/linux/utils.py:87

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1868125/+subscriptions