← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #93054

[Bug 2042938] [NEW] [OVN] Neutron Service Restart Disrupts Octavia OVN Load Balancer Floating IP

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Description:
Following the Neutron service restart, the Octavia OVN load balancer equipped with floating IPs stops processing traffic (DVR FIP is enabled)

Affected Versions:
- Neutron: Stable/Yoga (commit 03ed5578f7)
- Octavia: Stable/Yoga (commit 83955125)
- OVN: Version 22.12 (package ovn22.12-22.12.0-34.el9s.x86_64)

Details:
It looks like the 'external_mac' attribute for the Load Balancer's floating IP is changed when neutron is restarted.

here is a before neutron server restart:

ovn-nbctl find NAT type=dnat_and_snat external_ip=128.130.194.85
_uuid               : 438dff1d-3245-491c-a534-0a33e395ab2c
allowed_ext_ips     : []
exempted_ext_ips    : []
external_ids        : {"neutron:fip_external_mac"="fa:16:3e:cb:96:e7", "neutron:fip_id"="36fb8531-3a33-453e-be46-64599976aa0e", "neutron:fip_network_id"="1f115e41-cc7f-4bb5-8493-995d035277dc", "neutron:fip_port_id"="9ef678e5-add1-47f5-ac7c-a23f4ee753bb", "neutron:revision_number"="50", "neutron:router_name"=neutron-96bd0fd6-bc77-4b17-a5b4-e79c20b448ce}
external_ip         : "128.130.194.85"
external_mac        : []
external_port_range : ""
gateway_port        : []
logical_ip          : "192.168.100.124"
logical_port        : "9ef678e5-add1-47f5-ac7c-a23f4ee753bb"
options             : {}
type                : dnat_and_snat

after neutron server restart:

ovn-nbctl find NAT type=dnat_and_snat external_ip=128.130.194.8
_uuid               : 028a9017-544b-49e6-82cb-86cb1bdfab54
allowed_ext_ips     : []
exempted_ext_ips    : []
external_ids        : {"neutron:fip_external_mac"="fa:16:3e:f1:08:95", "neutron:fip_id"="3bbd5870-54ba-4941-a85a-29691d8b7aae", "neutron:fip_network_id"="1f115e41-cc7f-4bb5-8493-995d035277dc", "neutron:fip_port_id"="f2791464-91b6-4ace-8255-5e4aaf9785e0", "neutron:revision_number"="30", "neutron:router_name"=neutron-6d32d3de-38cc-4175-9764-5bdd1efbc20e}
external_ip         : "128.130.194.8"
external_mac        : "fa:16:3e:f1:08:95"
external_port_range : ""
gateway_port        : []
logical_ip          : "192.168.100.213"
logical_port        : "f2791464-91b6-4ace-8255-5e4aaf9785e0"
options             : {}
type                : dnat_and_snat

here are the debug logs entries for this FIP when neutron restarts:

2023-11-07 12:46:26.165 28 DEBUG ovsdbapp.backend.ovs_idl.event
[req-4ac48a1f-12b8-4d33-a05a-5ca856fe1895 - - - - -] Matched CREATE:
LogicalSwitchPortCreateDownEvent(events=('create',),
table='Logical_Switch_Port', conditions=(('up', '=', False),),
old_conditions=None), priority=20 to
row=Logical_Switch_Port(port_security=['fa:16:3e:54:5e:8c
192.168.3.236'], addresses=[], type=, dhcpv4_options=[<ovs.db.idl.Row
object at 0x7f1a36a172b0>], name=a803471b-0254-4f66-911e-03a143c874b8,
up=[False], options={'requested-chassis': ''}, ha_chassis_group=[],
external_ids={'neutron:cidrs': '192.168.3.236/24', 'neutron:device_id':
'', 'neutron:device_owner': '', 'neutron:network_name': 'neutron-
eb3ed2db-3c79-4e78-92ac-5653f9abb094', 'neutron:port_fip':
'128.130.194.113', 'neutron:port_name': 'ovn-lb-
vip-c60d029a-7625-4a96-837d-46ba95f56507', 'neutron:project_id':
'ea03412a87bc426eae8b7badd15aa7a8', 'neutron:revision_number': '1',
'neutron:security_group_ids': 'f558ec6a-fcb7-4297-8737-11a1507099b0',
'neutron:subnet_pool_addr_scope4': '',
'neutron:subnet_pool_addr_scope6': ''}, dynamic_addresses=[], tag=[],
parent_name=[], mirror_rules=[], tag_request=[], enabled=[True],
dhcpv6_options=[]) old= matches /var/lib/kolla/venv/lib/python3.9/site-
packages/ovsdbapp/backend/ovs_idl/event.py:43

2023-11-07 12:46:26.209 28 DEBUG ovsdbapp.backend.ovs_idl.event
[req-4ac48a1f-12b8-4d33-a05a-5ca856fe1895 - - - - -] Matched CREATE:
FIPAddDeleteEvent(events=('create', 'delete'), table='NAT',
conditions=(('type', '=', 'dnat_and_snat'),), old_conditions=None),
priority=20 to row=NAT(external_ids={'neutron:fip_external_mac':
'fa:16:3e:17:d1:2c', 'neutron:fip_id':
'840d24da-368d-4851-9b19-131c8f882f81', 'neutron:fip_network_id':
'1f115e41-cc7f-4bb5-8493-995d035277dc', 'neutron:fip_port_id':
'a803471b-0254-4f66-911e-03a143c874b8', 'neutron:revision_number': '8',
'neutron:router_name': 'neutron-83f83d78-b4a5-4fe0-8e32-352f6d941583'},
external_ip=128.130.194.113, allowed_ext_ips=[], external_port_range=,
exempted_ext_ips=[], logical_ip=192.168.3.236, type=dnat_and_snat,
external_mac=['fa:16:3e:17:d1:2c'], options={},
logical_port=['a803471b-0254-4f66-911e-03a143c874b8'], gateway_port=[])
old= matches /var/lib/kolla/venv/lib/python3.9/site-
packages/ovsdbapp/backend/ovs_idl/event.py:43

2023-11-07 12:46:26.225 31 DEBUG ovsdbapp.backend.ovs_idl.event
[req-985a5a2e-6838-45c1-9879-aa8bc0732d00 - - - - -] Matched CREATE:
FIPAddDeleteEvent(events=('create', 'delete'), table='NAT',
conditions=(('type', '=', 'dnat_and_snat'),), old_conditions=None),
priority=20 to row=NAT(external_ids={'neutron:fip_external_mac':
'fa:16:3e:17:d1:2c', 'neutron:fip_id':
'840d24da-368d-4851-9b19-131c8f882f81', 'neutron:fip_network_id':
'1f115e41-cc7f-4bb5-8493-995d035277dc', 'neutron:fip_port_id':
'a803471b-0254-4f66-911e-03a143c874b8', 'neutron:revision_number': '8',
'neutron:router_name': 'neutron-83f83d78-b4a5-4fe0-8e32-352f6d941583'},
external_ip=128.130.194.113, allowed_ext_ips=[], external_port_range=,
exempted_ext_ips=[], logical_ip=192.168.3.236, type=dnat_and_snat,
external_mac=['fa:16:3e:17:d1:2c'], options={},
logical_port=['a803471b-0254-4f66-911e-03a143c874b8'], gateway_port=[])
old= matches /var/lib/kolla/venv/lib/python3.9/site-
packages/ovsdbapp/backend/ovs_idl/event.py:43

2023-11-07 12:46:42.564 31 DEBUG oslo_concurrency.processutils
[req-e1654d04-1e37-44e2-a10b-16fc937ba532 - - - - -] Running cmd
(subprocess): ovsdb-client transact
tcp:10.20.1.11:6642,tcp:10.20.1.12:6642,tcp:10.20.1.13:6642 --timeout
180 ["OVN_Southbound", {"op": "delete", "table": "MAC_Binding", "where":
[["ip", "==", "128.130.194.113"]]}] execute
/var/lib/kolla/venv/lib/python3.9/site-
packages/oslo_concurrency/processutils.py:384

2023-11-07 12:46:42.575 31 DEBUG oslo_concurrency.processutils
[req-e1654d04-1e37-44e2-a10b-16fc937ba532 - - - - -] CMD "ovsdb-client
transact tcp:10.20.1.11:6642,tcp:10.20.1.12:6642,tcp:10.20.1.13:6642
--timeout 180 ["OVN_Southbound", {"op": "delete", "table":
"MAC_Binding", "where": [["ip", "==", "128.130.194.113"]]}]" returned: 0
in 0.011s execute /var/lib/kolla/venv/lib/python3.9/site-
packages/oslo_concurrency/processutils.py:422

2023-11-07 12:46:49.623 28 DEBUG oslo_concurrency.processutils
[req-c8da7baa-f23d-4a4d-aa5f-307ac4957ee6 - - - - -] Running cmd
(subprocess): ovsdb-client transact
tcp:10.20.1.11:6642,tcp:10.20.1.12:6642,tcp:10.20.1.13:6642 --timeout
180 ["OVN_Southbound", {"op": "delete", "table": "MAC_Binding", "where":
[["ip", "==", "128.130.194.113"]]}] execute
/var/lib/kolla/venv/lib/python3.9/site-
packages/oslo_concurrency/processutils.py:384

2023-11-07 12:46:49.633 28 DEBUG oslo_concurrency.processutils
[req-c8da7baa-f23d-4a4d-aa5f-307ac4957ee6 - - - - -] CMD "ovsdb-client
transact tcp:10.20.1.11:6642,tcp:10.20.1.12:6642,tcp:10.20.1.13:6642
--timeout 180 ["OVN_Southbound", {"op": "delete", "table":
"MAC_Binding", "where": [["ip", "==", "128.130.194.113"]]}]" returned: 0
in 0.010s execute /var/lib/kolla/venv/lib/python3.9/site-
packages/oslo_concurrency/processutils.py:422

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2042938

Title:
  [OVN] Neutron Service Restart Disrupts Octavia OVN Load Balancer
  Floating IP

Status in neutron:
  New

Bug description:
  Description:
  Following the Neutron service restart, the Octavia OVN load balancer equipped with floating IPs stops processing traffic (DVR FIP is enabled)

  Affected Versions:
  - Neutron: Stable/Yoga (commit 03ed5578f7)
  - Octavia: Stable/Yoga (commit 83955125)
  - OVN: Version 22.12 (package ovn22.12-22.12.0-34.el9s.x86_64)

  Details:
  It looks like the 'external_mac' attribute for the Load Balancer's floating IP is changed when neutron is restarted.

  here is a before neutron server restart:

  ovn-nbctl find NAT type=dnat_and_snat external_ip=128.130.194.85
  _uuid               : 438dff1d-3245-491c-a534-0a33e395ab2c
  allowed_ext_ips     : []
  exempted_ext_ips    : []
  external_ids        : {"neutron:fip_external_mac"="fa:16:3e:cb:96:e7", "neutron:fip_id"="36fb8531-3a33-453e-be46-64599976aa0e", "neutron:fip_network_id"="1f115e41-cc7f-4bb5-8493-995d035277dc", "neutron:fip_port_id"="9ef678e5-add1-47f5-ac7c-a23f4ee753bb", "neutron:revision_number"="50", "neutron:router_name"=neutron-96bd0fd6-bc77-4b17-a5b4-e79c20b448ce}
  external_ip         : "128.130.194.85"
  external_mac        : []
  external_port_range : ""
  gateway_port        : []
  logical_ip          : "192.168.100.124"
  logical_port        : "9ef678e5-add1-47f5-ac7c-a23f4ee753bb"
  options             : {}
  type                : dnat_and_snat

  after neutron server restart:

  ovn-nbctl find NAT type=dnat_and_snat external_ip=128.130.194.8
  _uuid               : 028a9017-544b-49e6-82cb-86cb1bdfab54
  allowed_ext_ips     : []
  exempted_ext_ips    : []
  external_ids        : {"neutron:fip_external_mac"="fa:16:3e:f1:08:95", "neutron:fip_id"="3bbd5870-54ba-4941-a85a-29691d8b7aae", "neutron:fip_network_id"="1f115e41-cc7f-4bb5-8493-995d035277dc", "neutron:fip_port_id"="f2791464-91b6-4ace-8255-5e4aaf9785e0", "neutron:revision_number"="30", "neutron:router_name"=neutron-6d32d3de-38cc-4175-9764-5bdd1efbc20e}
  external_ip         : "128.130.194.8"
  external_mac        : "fa:16:3e:f1:08:95"
  external_port_range : ""
  gateway_port        : []
  logical_ip          : "192.168.100.213"
  logical_port        : "f2791464-91b6-4ace-8255-5e4aaf9785e0"
  options             : {}
  type                : dnat_and_snat

  here are the debug logs entries for this FIP when neutron restarts:

  2023-11-07 12:46:26.165 28 DEBUG ovsdbapp.backend.ovs_idl.event
  [req-4ac48a1f-12b8-4d33-a05a-5ca856fe1895 - - - - -] Matched CREATE:
  LogicalSwitchPortCreateDownEvent(events=('create',),
  table='Logical_Switch_Port', conditions=(('up', '=', False),),
  old_conditions=None), priority=20 to
  row=Logical_Switch_Port(port_security=['fa:16:3e:54:5e:8c
  192.168.3.236'], addresses=[], type=, dhcpv4_options=[<ovs.db.idl.Row
  object at 0x7f1a36a172b0>], name=a803471b-0254-4f66-911e-03a143c874b8,
  up=[False], options={'requested-chassis': ''}, ha_chassis_group=[],
  external_ids={'neutron:cidrs': '192.168.3.236/24',
  'neutron:device_id': '', 'neutron:device_owner': '',
  'neutron:network_name': 'neutron-
  eb3ed2db-3c79-4e78-92ac-5653f9abb094', 'neutron:port_fip':
  '128.130.194.113', 'neutron:port_name': 'ovn-lb-
  vip-c60d029a-7625-4a96-837d-46ba95f56507', 'neutron:project_id':
  'ea03412a87bc426eae8b7badd15aa7a8', 'neutron:revision_number': '1',
  'neutron:security_group_ids': 'f558ec6a-fcb7-4297-8737-11a1507099b0',
  'neutron:subnet_pool_addr_scope4': '',
  'neutron:subnet_pool_addr_scope6': ''}, dynamic_addresses=[], tag=[],
  parent_name=[], mirror_rules=[], tag_request=[], enabled=[True],
  dhcpv6_options=[]) old= matches
  /var/lib/kolla/venv/lib/python3.9/site-
  packages/ovsdbapp/backend/ovs_idl/event.py:43

  2023-11-07 12:46:26.209 28 DEBUG ovsdbapp.backend.ovs_idl.event
  [req-4ac48a1f-12b8-4d33-a05a-5ca856fe1895 - - - - -] Matched CREATE:
  FIPAddDeleteEvent(events=('create', 'delete'), table='NAT',
  conditions=(('type', '=', 'dnat_and_snat'),), old_conditions=None),
  priority=20 to row=NAT(external_ids={'neutron:fip_external_mac':
  'fa:16:3e:17:d1:2c', 'neutron:fip_id':
  '840d24da-368d-4851-9b19-131c8f882f81', 'neutron:fip_network_id':
  '1f115e41-cc7f-4bb5-8493-995d035277dc', 'neutron:fip_port_id':
  'a803471b-0254-4f66-911e-03a143c874b8', 'neutron:revision_number':
  '8', 'neutron:router_name':
  'neutron-83f83d78-b4a5-4fe0-8e32-352f6d941583'},
  external_ip=128.130.194.113, allowed_ext_ips=[], external_port_range=,
  exempted_ext_ips=[], logical_ip=192.168.3.236, type=dnat_and_snat,
  external_mac=['fa:16:3e:17:d1:2c'], options={},
  logical_port=['a803471b-0254-4f66-911e-03a143c874b8'],
  gateway_port=[]) old= matches /var/lib/kolla/venv/lib/python3.9/site-
  packages/ovsdbapp/backend/ovs_idl/event.py:43

  2023-11-07 12:46:26.225 31 DEBUG ovsdbapp.backend.ovs_idl.event
  [req-985a5a2e-6838-45c1-9879-aa8bc0732d00 - - - - -] Matched CREATE:
  FIPAddDeleteEvent(events=('create', 'delete'), table='NAT',
  conditions=(('type', '=', 'dnat_and_snat'),), old_conditions=None),
  priority=20 to row=NAT(external_ids={'neutron:fip_external_mac':
  'fa:16:3e:17:d1:2c', 'neutron:fip_id':
  '840d24da-368d-4851-9b19-131c8f882f81', 'neutron:fip_network_id':
  '1f115e41-cc7f-4bb5-8493-995d035277dc', 'neutron:fip_port_id':
  'a803471b-0254-4f66-911e-03a143c874b8', 'neutron:revision_number':
  '8', 'neutron:router_name':
  'neutron-83f83d78-b4a5-4fe0-8e32-352f6d941583'},
  external_ip=128.130.194.113, allowed_ext_ips=[], external_port_range=,
  exempted_ext_ips=[], logical_ip=192.168.3.236, type=dnat_and_snat,
  external_mac=['fa:16:3e:17:d1:2c'], options={},
  logical_port=['a803471b-0254-4f66-911e-03a143c874b8'],
  gateway_port=[]) old= matches /var/lib/kolla/venv/lib/python3.9/site-
  packages/ovsdbapp/backend/ovs_idl/event.py:43

  2023-11-07 12:46:42.564 31 DEBUG oslo_concurrency.processutils
  [req-e1654d04-1e37-44e2-a10b-16fc937ba532 - - - - -] Running cmd
  (subprocess): ovsdb-client transact
  tcp:10.20.1.11:6642,tcp:10.20.1.12:6642,tcp:10.20.1.13:6642 --timeout
  180 ["OVN_Southbound", {"op": "delete", "table": "MAC_Binding",
  "where": [["ip", "==", "128.130.194.113"]]}] execute
  /var/lib/kolla/venv/lib/python3.9/site-
  packages/oslo_concurrency/processutils.py:384

  2023-11-07 12:46:42.575 31 DEBUG oslo_concurrency.processutils
  [req-e1654d04-1e37-44e2-a10b-16fc937ba532 - - - - -] CMD "ovsdb-client
  transact tcp:10.20.1.11:6642,tcp:10.20.1.12:6642,tcp:10.20.1.13:6642
  --timeout 180 ["OVN_Southbound", {"op": "delete", "table":
  "MAC_Binding", "where": [["ip", "==", "128.130.194.113"]]}]" returned:
  0 in 0.011s execute /var/lib/kolla/venv/lib/python3.9/site-
  packages/oslo_concurrency/processutils.py:422

  2023-11-07 12:46:49.623 28 DEBUG oslo_concurrency.processutils
  [req-c8da7baa-f23d-4a4d-aa5f-307ac4957ee6 - - - - -] Running cmd
  (subprocess): ovsdb-client transact
  tcp:10.20.1.11:6642,tcp:10.20.1.12:6642,tcp:10.20.1.13:6642 --timeout
  180 ["OVN_Southbound", {"op": "delete", "table": "MAC_Binding",
  "where": [["ip", "==", "128.130.194.113"]]}] execute
  /var/lib/kolla/venv/lib/python3.9/site-
  packages/oslo_concurrency/processutils.py:384

  2023-11-07 12:46:49.633 28 DEBUG oslo_concurrency.processutils
  [req-c8da7baa-f23d-4a4d-aa5f-307ac4957ee6 - - - - -] CMD "ovsdb-client
  transact tcp:10.20.1.11:6642,tcp:10.20.1.12:6642,tcp:10.20.1.13:6642
  --timeout 180 ["OVN_Southbound", {"op": "delete", "table":
  "MAC_Binding", "where": [["ip", "==", "128.130.194.113"]]}]" returned:
  0 in 0.010s execute /var/lib/kolla/venv/lib/python3.9/site-
  packages/oslo_concurrency/processutils.py:422

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2042938/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next