yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1987530] [NEW] Duplicate external_ip in NAT table lead to loss of N/S connectivity

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Frode Nordahl <1987530@xxxxxxxxxxxxxxxxxx>
Date: Wed, 24 Aug 2022 14:00:16 -0000
Reply-to: Bug 1987530 <1987530@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

In a busy CI cloud Neutron appears to leave behind stale NAT records.
When this happens a new instance may come around referencing the same
external_ip, and the new instance will have connectivity issues.

An example showing this situation:
$ sudo ovn-nbctl list nat|grep -B5 -A5 10.245.165.87

_uuid               : 97e89f15-007e-4cf6-865f-9c796e11faf5
allowed_ext_ips     : []
exempted_ext_ips    : []
external_ids        : {"neutron:fip_external_mac"="fa:16:3e:7e:c2:a1", "neutron:fip_id"="be6fa0b0-458c-42a1-b682-a4d6c33982be", "neutron:fip_port_id"="85bf473a-eafe-4ff1-85e3-810cad395295", "neutron:revision_number"="0", "neutron:router_name"=neutron-bfcd42ec-abb3-4a8b-8774-42767cc34cfb}
external_ip         : "10.245.165.87"
external_mac        : []
external_port_range : ""
logical_ip          : "172.16.0.90"
logical_port        : "85bf473a-eafe-4ff1-85e3-810cad395295"
options             : {}
--

_uuid               : 2dc59d66-8866-47d4-a97d-17ee5b8874e2
allowed_ext_ips     : []
exempted_ext_ips    : []
external_ids        : {"neutron:fip_external_mac"="fa:16:3e:28:a6:a8", "neutron:fip_id"="4f9524c6-5b99-45bf-b4bf-8f840231f19e", "neutron:fip_network_id"="5827fa17-882a-4c19-89a8-f6fe4c28d6dd", "neutron:fip_port_id"="08adde8a-60d4-4ed0-b08b-733ba12ad2ce", "neutron:revision_number"="6", "neutron:router_name"=neutron-3824b194-35c2-4511-95c5-2ab5b174c324}
external_ip         : "10.245.165.87"
external_mac        : []
external_port_range : ""
logical_ip          : "10.5.0.26"
logical_port        : "08adde8a-60d4-4ed0-b08b-733ba12ad2ce"
options             : {stateless="true"}

So the questions then become:
1) Are there anything to de done with the OVN data structure to prevent this from happening.
2) What can Neutron do to not leave these records behind and/or clean them up.

** Affects: neutron
     Importance: Undecided
         Status: New

** Affects: ovn (Ubuntu)
     Importance: Undecided
         Status: New

** Also affects: neutron
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1987530

Title:
  Duplicate external_ip in NAT table lead to loss of N/S connectivity

Status in neutron:
  New
Status in ovn package in Ubuntu:
  New

Bug description:
  In a busy CI cloud Neutron appears to leave behind stale NAT records.
  When this happens a new instance may come around referencing the same
  external_ip, and the new instance will have connectivity issues.

  An example showing this situation:
  $ sudo ovn-nbctl list nat|grep -B5 -A5 10.245.165.87

  _uuid               : 97e89f15-007e-4cf6-865f-9c796e11faf5
  allowed_ext_ips     : []
  exempted_ext_ips    : []
  external_ids        : {"neutron:fip_external_mac"="fa:16:3e:7e:c2:a1", "neutron:fip_id"="be6fa0b0-458c-42a1-b682-a4d6c33982be", "neutron:fip_port_id"="85bf473a-eafe-4ff1-85e3-810cad395295", "neutron:revision_number"="0", "neutron:router_name"=neutron-bfcd42ec-abb3-4a8b-8774-42767cc34cfb}
  external_ip         : "10.245.165.87"
  external_mac        : []
  external_port_range : ""
  logical_ip          : "172.16.0.90"
  logical_port        : "85bf473a-eafe-4ff1-85e3-810cad395295"
  options             : {}
  --

  _uuid               : 2dc59d66-8866-47d4-a97d-17ee5b8874e2
  allowed_ext_ips     : []
  exempted_ext_ips    : []
  external_ids        : {"neutron:fip_external_mac"="fa:16:3e:28:a6:a8", "neutron:fip_id"="4f9524c6-5b99-45bf-b4bf-8f840231f19e", "neutron:fip_network_id"="5827fa17-882a-4c19-89a8-f6fe4c28d6dd", "neutron:fip_port_id"="08adde8a-60d4-4ed0-b08b-733ba12ad2ce", "neutron:revision_number"="6", "neutron:router_name"=neutron-3824b194-35c2-4511-95c5-2ab5b174c324}
  external_ip         : "10.245.165.87"
  external_mac        : []
  external_port_range : ""
  logical_ip          : "10.5.0.26"
  logical_port        : "08adde8a-60d4-4ed0-b08b-733ba12ad2ce"
  options             : {stateless="true"}

  So the questions then become:
  1) Are there anything to de done with the OVN data structure to prevent this from happening.
  2) What can Neutron do to not leave these records behind and/or clean them up.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1987530/+subscriptions