yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1873761] [NEW] Internal IP leak to physical interface from qrouter in DVR mode

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Yedhu Sastri <1873761@xxxxxxxxxxxxxxxxxx>
Date: Mon, 20 Apr 2020 08:41:54 -0000
Reply-to: Bug 1873761 <1873761@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Setup: Openstack-Ansible cluster(Rocky - 18.1.8) with computes nodes using DVR. OS version Ubuntu 16.04.6 LTS with kernel 4.15.0-34-generic.
 
Problem: We can see internal IP leaked without NAT on our physical interface. This happens in TCP communication where client stopped abruptly before the server. 

Steps to reproduce:

TCP Client(192.168.100.24, 10.96.48.159)  
TCP Server(192.168.100.20, 10.96.48.207)

Server sends RST packets on connection termination.

Step1: Start the server and client.
Setp2: Stop the client(KeyboardInterrupt) while the server is still in the connection. 

tcpdump on the bond interface of the compute node in which the tcp
client is running

07:50:35.658208 IP 10.96.48.159.36394 > 10.96.48.207.5005: Flags [S], seq 3764020836, win 64240, options [mss 1460,sackOK,TS val 2823050719 ecr 0,nop,wscale 7], length 0
07:50:35.658539 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [S.], seq 1750463809, ack 3764020837, win 65160, options [mss 1460,sackOK,TS val 2874529221 ecr 2823050719,nop,wscale 7], length 0
07:50:35.658717 IP 10.96.48.159.36394 > 10.96.48.207.5005: Flags [.], ack 1, win 502, options [nop,nop,TS val 2823050720 ecr 2874529221], length 0
07:50:35.658746 IP 10.96.48.159.36394 > 10.96.48.207.5005: Flags [P.], seq 1:14, ack 1, win 502, options [nop,nop,TS val 2823050720 ecr 2874529221], length 13
07:50:35.658949 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [.], ack 14, win 509, options [nop,nop,TS val 2874529221 ecr 2823050720], length 0
07:50:35.659113 IP 10.96.48.159.36394 > 10.96.48.207.5005: Flags [P.], seq 14:32, ack 1, win 502, options [nop,nop,TS val 2823050720 ecr 2874529221], length 18
07:50:35.659299 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [.], ack 32, win 509, options [nop,nop,TS val 2874529221 ecr 2823050720], length 0
07:50:40.729542 IP 10.96.48.159.36394 > 10.96.48.207.5005: Flags [F.], seq 32, ack 1, win 502, options [nop,nop,TS val 2823055790 ecr 2874529221], length 0
07:50:40.773484 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [.], ack 33, win 509, options [nop,nop,TS val 2874534335 ecr 2823055790], length 0
07:53:35.732815 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [P.], seq 1:21, ack 33, win 509, options [nop,nop,TS val 2874709290 ecr 2823055790], length 20
07:53:35.732878 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [R.], seq 21, ack 33, win 509, options [nop,nop,TS val 2874709291 ecr 2823055790], length 0

07:53:35.733668 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R],
seq 3764020869, win 0, length 0


tcpdump on the bond interface of the compute node in which the tcp server is running


07:50:35.658302 IP 10.96.48.159.36394 > 10.96.48.207.5005: Flags [S], seq 3764020836, win 64240, options [mss 1460,sackOK,TS val 2823050719 ecr 0,nop,wscale 7], length 0
07:50:35.658589 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [S.], seq 1750463809, ack 3764020837, win 65160, options [mss 1460,sackOK,TS val 2874529221 ecr 2823050719,nop,wscale 7], length 0
07:50:35.658811 IP 10.96.48.159.36394 > 10.96.48.207.5005: Flags [.], ack 1, win 502, options [nop,nop,TS val 2823050720 ecr 2874529221], length 0
07:50:35.658901 IP 10.96.48.159.36394 > 10.96.48.207.5005: Flags [P.], seq 1:14, ack 1, win 502, options [nop,nop,TS val 2823050720 ecr 2874529221], length 13
07:50:35.658998 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [.], ack 14, win 509, options [nop,nop,TS val 2874529221 ecr 2823050720], length 0
07:50:35.659205 IP 10.96.48.159.36394 > 10.96.48.207.5005: Flags [P.], seq 14:32, ack 1, win 502, options [nop,nop,TS val 2823050720 ecr 2874529221], length 18
07:50:35.659350 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [.], ack 32, win 509, options [nop,nop,TS val 2874529221 ecr 2823050720], length 0
07:50:40.729633 IP 10.96.48.159.36394 > 10.96.48.207.5005: Flags [F.], seq 32, ack 1, win 502, options [nop,nop,TS val 2823055790 ecr 2874529221], length 0
07:50:40.773533 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [.], ack 33, win 509, options [nop,nop,TS val 2874534335 ecr 2823055790], length 0
07:53:35.732868 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [P.], seq 1:21, ack 33, win 509, options [nop,nop,TS val 2874709290 ecr 2823055790], length 20
07:53:35.732898 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [R.], seq 21, ack 33, win 509, options [nop,nop,TS val 2874709291 ecr 2823055790], length 0

07:53:35.733767 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.734408 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.734602 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.734748 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.734873 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.734973 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.735073 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.735171 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.735269 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.735366 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.735464 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.735561 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.735662 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.735776 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.735877 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.735975 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.736073 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.736171 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.736269 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.736367 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.736465 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1873761

Title:
  Internal IP leak to physical interface from qrouter in DVR mode

Status in neutron:
  New

Bug description:
  Setup: Openstack-Ansible cluster(Rocky - 18.1.8) with computes nodes using DVR. OS version Ubuntu 16.04.6 LTS with kernel 4.15.0-34-generic.
   
  Problem: We can see internal IP leaked without NAT on our physical interface. This happens in TCP communication where client stopped abruptly before the server. 

  Steps to reproduce:

  TCP Client(192.168.100.24, 10.96.48.159)  
  TCP Server(192.168.100.20, 10.96.48.207)

  Server sends RST packets on connection termination.

  Step1: Start the server and client.
  Setp2: Stop the client(KeyboardInterrupt) while the server is still in the connection. 

  tcpdump on the bond interface of the compute node in which the tcp
  client is running

  07:50:35.658208 IP 10.96.48.159.36394 > 10.96.48.207.5005: Flags [S], seq 3764020836, win 64240, options [mss 1460,sackOK,TS val 2823050719 ecr 0,nop,wscale 7], length 0
  07:50:35.658539 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [S.], seq 1750463809, ack 3764020837, win 65160, options [mss 1460,sackOK,TS val 2874529221 ecr 2823050719,nop,wscale 7], length 0
  07:50:35.658717 IP 10.96.48.159.36394 > 10.96.48.207.5005: Flags [.], ack 1, win 502, options [nop,nop,TS val 2823050720 ecr 2874529221], length 0
  07:50:35.658746 IP 10.96.48.159.36394 > 10.96.48.207.5005: Flags [P.], seq 1:14, ack 1, win 502, options [nop,nop,TS val 2823050720 ecr 2874529221], length 13
  07:50:35.658949 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [.], ack 14, win 509, options [nop,nop,TS val 2874529221 ecr 2823050720], length 0
  07:50:35.659113 IP 10.96.48.159.36394 > 10.96.48.207.5005: Flags [P.], seq 14:32, ack 1, win 502, options [nop,nop,TS val 2823050720 ecr 2874529221], length 18
  07:50:35.659299 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [.], ack 32, win 509, options [nop,nop,TS val 2874529221 ecr 2823050720], length 0
  07:50:40.729542 IP 10.96.48.159.36394 > 10.96.48.207.5005: Flags [F.], seq 32, ack 1, win 502, options [nop,nop,TS val 2823055790 ecr 2874529221], length 0
  07:50:40.773484 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [.], ack 33, win 509, options [nop,nop,TS val 2874534335 ecr 2823055790], length 0
  07:53:35.732815 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [P.], seq 1:21, ack 33, win 509, options [nop,nop,TS val 2874709290 ecr 2823055790], length 20
  07:53:35.732878 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [R.], seq 21, ack 33, win 509, options [nop,nop,TS val 2874709291 ecr 2823055790], length 0

  07:53:35.733668 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags
  [R], seq 3764020869, win 0, length 0

  
  tcpdump on the bond interface of the compute node in which the tcp server is running

  
  07:50:35.658302 IP 10.96.48.159.36394 > 10.96.48.207.5005: Flags [S], seq 3764020836, win 64240, options [mss 1460,sackOK,TS val 2823050719 ecr 0,nop,wscale 7], length 0
  07:50:35.658589 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [S.], seq 1750463809, ack 3764020837, win 65160, options [mss 1460,sackOK,TS val 2874529221 ecr 2823050719,nop,wscale 7], length 0
  07:50:35.658811 IP 10.96.48.159.36394 > 10.96.48.207.5005: Flags [.], ack 1, win 502, options [nop,nop,TS val 2823050720 ecr 2874529221], length 0
  07:50:35.658901 IP 10.96.48.159.36394 > 10.96.48.207.5005: Flags [P.], seq 1:14, ack 1, win 502, options [nop,nop,TS val 2823050720 ecr 2874529221], length 13
  07:50:35.658998 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [.], ack 14, win 509, options [nop,nop,TS val 2874529221 ecr 2823050720], length 0
  07:50:35.659205 IP 10.96.48.159.36394 > 10.96.48.207.5005: Flags [P.], seq 14:32, ack 1, win 502, options [nop,nop,TS val 2823050720 ecr 2874529221], length 18
  07:50:35.659350 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [.], ack 32, win 509, options [nop,nop,TS val 2874529221 ecr 2823050720], length 0
  07:50:40.729633 IP 10.96.48.159.36394 > 10.96.48.207.5005: Flags [F.], seq 32, ack 1, win 502, options [nop,nop,TS val 2823055790 ecr 2874529221], length 0
  07:50:40.773533 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [.], ack 33, win 509, options [nop,nop,TS val 2874534335 ecr 2823055790], length 0
  07:53:35.732868 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [P.], seq 1:21, ack 33, win 509, options [nop,nop,TS val 2874709290 ecr 2823055790], length 20
  07:53:35.732898 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [R.], seq 21, ack 33, win 509, options [nop,nop,TS val 2874709291 ecr 2823055790], length 0

  07:53:35.733767 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
  07:53:35.734408 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
  07:53:35.734602 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
  07:53:35.734748 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
  07:53:35.734873 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
  07:53:35.734973 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
  07:53:35.735073 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
  07:53:35.735171 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
  07:53:35.735269 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
  07:53:35.735366 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
  07:53:35.735464 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
  07:53:35.735561 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
  07:53:35.735662 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
  07:53:35.735776 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
  07:53:35.735877 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
  07:53:35.735975 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
  07:53:35.736073 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
  07:53:35.736171 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
  07:53:35.736269 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
  07:53:35.736367 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
  07:53:35.736465 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1873761/+subscriptions