← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #77742

[Bug 1822256] [NEW] Ip fragments lost when restart ovs-agent with openvswitch firewall

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

environment：
linux version: Linux controller.novalocal 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
OpenStack version: Rocky
network type: vxlan or vlan
firewall driver: openvswitch

1. Create 2 VMs(vm1, vm2) in different compute nodes(node-1, node-2)
with all tcp passed sg in one network.

2. Login to vm2, create a large file, for example:
vm2# dd if=/dev/zero of=/mnt/test.img bs=1G count=5

3.Login to vm1, scp vm2's large file into vm1, when scp process starts, go to step 4.
vm1# scp vm2-ip:/mnt/test.img /mnt

4.Login to node-2, and restart neutron-openvswitch-agent, this will refresh all the openflow in br-int
node-2# systemctl restart neutron-openvswitch-agent

5.Login to vm1, and after several seconds, you will find the scp process
status is stalled.

After some investigation, I found the openflow refresh causes ip
fragments lost.When this happened, I captured packets with "tcpdump -i
tap-xxx -w tmp.pcap", and with wireshark I saw these errors:

192.168.100.19	192.168.100.5	SSH	16478	Server: [TCP ACKed unseen segment] [TCP Previous segment not captured] , Encrypted packet (len=16412)
192.168.100.19	192.168.100.5	SSH	8302	Server: [TCP ACKed unseen segment] , Encrypted packet (len=8236)
192.168.100.5	192.168.100.19	TCP	66	[TCP ACKed unseen segment] [TCP Previous segment not captured] 54354 → 22 [ACK] Seq=2509 Ack=600733 Win=16522 Len=0 TSval=2847412 TSecr=2851031
192.168.100.19	192.168.100.5	SSH	1464	Server: [TCP Spurious Retransmission] , Encrypted packet (len=1398)
192.168.100.5	192.168.100.19	TCP	78	[TCP Dup ACK 25182#1] 54354 → 22 [ACK] Seq=326305 Ack=67089901 Win=18494 Len=0 TSval=2849742 TSecr=2853310 SLE=67073429 SRE=67074827
192.168.100.5	192.168.100.19	TCP	110	[TCP Retransmission] 54354 → 22 [PSH, ACK] Seq=326173 Ack=67089901 Win=18494 Len=44 TSval=2849742 TSecr=2853310

192.168.100.19	192.168.100.5	TCP	1464	[TCP Retransmission] 22 → 54354 [ACK] Seq=70971905 Ack=346105 Win=2016 Len=1398 TSval=2853361 TSecr=2849691
192.168.100.19	192.168.100.5	TCP	1464	[TCP Retransmission] 22 → 54354 [ACK] Seq=70971905 Ack=346105 Win=2016 Len=1398 TSval=2853463 TSecr=2849691
192.168.100.19	192.168.100.5	TCP	1464	[TCP Retransmission] 22 → 54354 [ACK] Seq=70971905 Ack=346105 Win=2016 Len=1398 TSval=2854076 TSecr=2849691

And I checked the statue of this tcp connect in both compute nodes, it's still ESTABLISHED.
# conntrack -L | grep 192.168.100.5
tcp      6 299 ESTABLISHED src=192.168.100.5 dst=192.168.100.19 sport=54356 dport=22 src=192.168.100.19 dst=192.168.100.5 sport=22 dport=54354 [ASSURED] mark=0 zone=4 use=1
#  conntrack -L | grep 192.168.100.5
tcp      6 287 ESTABLISHED src=192.168.100.5 dst=192.168.100.19 sport=54356 dport=22 src=192.168.100.19 dst=192.168.100.5 sport=22 dport=54354 [ASSURED] mark=0 zone=1 use=1

I have no idea why refresh openflow will cause ip fragments lost, hopes
someone has a way to solve this problem.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1822256

Title:
  Ip fragments lost when restart ovs-agent with openvswitch firewall

Status in neutron:
  New

Bug description:
  environment：
  linux version: Linux controller.novalocal 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
  OpenStack version: Rocky
  network type: vxlan or vlan
  firewall driver: openvswitch

  1. Create 2 VMs(vm1, vm2) in different compute nodes(node-1, node-2)
  with all tcp passed sg in one network.

  2. Login to vm2, create a large file, for example:
  vm2# dd if=/dev/zero of=/mnt/test.img bs=1G count=5

  3.Login to vm1, scp vm2's large file into vm1, when scp process starts, go to step 4.
  vm1# scp vm2-ip:/mnt/test.img /mnt

  4.Login to node-2, and restart neutron-openvswitch-agent, this will refresh all the openflow in br-int
  node-2# systemctl restart neutron-openvswitch-agent

  5.Login to vm1, and after several seconds, you will find the scp
  process status is stalled.

  After some investigation, I found the openflow refresh causes ip
  fragments lost.When this happened, I captured packets with "tcpdump -i
  tap-xxx -w tmp.pcap", and with wireshark I saw these errors:

  192.168.100.19	192.168.100.5	SSH	16478	Server: [TCP ACKed unseen segment] [TCP Previous segment not captured] , Encrypted packet (len=16412)
  192.168.100.19	192.168.100.5	SSH	8302	Server: [TCP ACKed unseen segment] , Encrypted packet (len=8236)
  192.168.100.5	192.168.100.19	TCP	66	[TCP ACKed unseen segment] [TCP Previous segment not captured] 54354 → 22 [ACK] Seq=2509 Ack=600733 Win=16522 Len=0 TSval=2847412 TSecr=2851031
  192.168.100.19	192.168.100.5	SSH	1464	Server: [TCP Spurious Retransmission] , Encrypted packet (len=1398)
  192.168.100.5	192.168.100.19	TCP	78	[TCP Dup ACK 25182#1] 54354 → 22 [ACK] Seq=326305 Ack=67089901 Win=18494 Len=0 TSval=2849742 TSecr=2853310 SLE=67073429 SRE=67074827
  192.168.100.5	192.168.100.19	TCP	110	[TCP Retransmission] 54354 → 22 [PSH, ACK] Seq=326173 Ack=67089901 Win=18494 Len=44 TSval=2849742 TSecr=2853310

  192.168.100.19	192.168.100.5	TCP	1464	[TCP Retransmission] 22 → 54354 [ACK] Seq=70971905 Ack=346105 Win=2016 Len=1398 TSval=2853361 TSecr=2849691
  192.168.100.19	192.168.100.5	TCP	1464	[TCP Retransmission] 22 → 54354 [ACK] Seq=70971905 Ack=346105 Win=2016 Len=1398 TSval=2853463 TSecr=2849691
  192.168.100.19	192.168.100.5	TCP	1464	[TCP Retransmission] 22 → 54354 [ACK] Seq=70971905 Ack=346105 Win=2016 Len=1398 TSval=2854076 TSecr=2849691

  And I checked the statue of this tcp connect in both compute nodes, it's still ESTABLISHED.
  # conntrack -L | grep 192.168.100.5
  tcp      6 299 ESTABLISHED src=192.168.100.5 dst=192.168.100.19 sport=54356 dport=22 src=192.168.100.19 dst=192.168.100.5 sport=22 dport=54354 [ASSURED] mark=0 zone=4 use=1
  #  conntrack -L | grep 192.168.100.5
  tcp      6 287 ESTABLISHED src=192.168.100.5 dst=192.168.100.19 sport=54356 dport=22 src=192.168.100.19 dst=192.168.100.5 sport=22 dport=54354 [ASSURED] mark=0 zone=1 use=1

  I have no idea why refresh openflow will cause ip fragments lost,
  hopes someone has a way to solve this problem.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1822256/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next