yahoo-eng-team team mailing list archive

[Eugene Nikanorov <enikanorov@xxxxxxxxxxxx>]

Removing duplicate to perform additional checks


** This bug is no longer a duplicate of bug 1478925
   Instances on the same compute node unable to connect to each other's ports

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1514017

Title:
  Sec Group IPtables dropping all TCP ACK as Invalid for all connections
  between instances on different networks on same libvirt host.

Status in neutron:
  New

Bug description:
  Sec Group rules are set to allow all TCP connections in both
  directions.

  When 2 instance on the same host but in different networks try to
  start a TCP session, the TCP handshake never completes as the ACK
  packet gets dropped as invalid by the security group iptable rules on
  the Linux bridge. The SYN and SYN_ACK packets get through. This
  happens on all libvirt hosts, when the 2 instances are the same host.
  When the instance are on separate hosts, the ACK packet gets through
  and the session is established.

  VM1 --> SYN --> VM2
  VM1 <-- SYN_ACK <-- VM2
  VM1 --> ACK --> ???

  In the case of the above running tcpdump on the host on the VM1 tap
  interface shows the ACK packet entering the bridge, a tcpdump on qvb
  interface on the far side of the bridge show the packet missing.

  The following is a sample tcpdump from the client tap interface
  showing SYN, SYN_ACK and ACK packets.

  12:08:34.685284 fa:16:3e:6d:ff:9f (oui Unknown) > 00:00:0c:07:ac:47 (oui Cisco), ethertype IPv4 (0x0800), length 74: 10.0.116.15.58374 > 10.0.115.11.ssh: Flags [S], seq 942942122, win 14600, options [mss 1460,sackOK,TS val 1211694066 ecr 0,nop,wscale 6], length 0
  12:08:34.686254 a2:30:cf:00:00:1e (oui Unknown) > fa:16:3e:6d:ff:9f (oui Unknown), ethertype IPv4 (0x0800), length 74: 10.0.115.11.ssh > 10.0.116.15.58374: Flags [S.], seq 2764187879, ack 942942123, win 5792, options [mss 1380,sackOK,TS val 1212626150 ecr 1211694066,nop,wscale 7], length 0
  12:08:34.686425 fa:16:3e:6d:ff:9f (oui Unknown) > 00:00:0c:07:ac:47 (oui Cisco), ethertype IPv4 (0x0800), length 66: 10.0.116.15.58374 > 10.0.115.11.ssh: Flags [.], ack 1, win 229, options [nop,nop,TS val 1211694067 ecr 1212626150], length 0
  12:08:38.096191 a2:30:cf:00:00:1e (oui Unknown) > fa:16:3e:6d:ff:9f (oui Unknown), ethertype IPv4 (0x0800), length 74: 10.0.115.11.ssh > 10.0.116.15.58374: Flags [S.], seq 2764187879, ack 942942123, win 5792, options [mss 1380,sackOK,TS val 1212629559 ecr 1211694066,nop,wscale 7], length 0
  12:08:38.096379 fa:16:3e:6d:ff:9f (oui Unknown) > 00:00:0c:07:ac:47 (oui Cisco), ethertype IPv4 (0x0800), length 66: 10.0.116.15.58374 > 10.0.115.11.ssh: Flags [.], ack 1, win 229, options [nop,nop,TS val 1211697477 ecr 1212626150], length 0

  After examining iptable rule counters while attempting to open a TCP
  session, the following rule in the neutron-openvswi-od080d3d6-c chain
  is dropping the packets.

  DROP       all  --  anywhere             anywhere             state
  INVALID /* Drop packets that appear related to an existing connection
  (e.g. TCP ACK/FIN) but do not have an entry in conntrack. */

  conntrack shows the TCP session as being in the correct SYN_RECV
  state.

  conntrack -L -s 10.0.116.15 -d 10.0.115.11 -p tcp
  tcp      6 57 SYN_RECV src=10.0.116.15 dst=10.0.115.11 sport=58619 dport=22 src=10.0.115.11 dst=10.0.116.15 sport=22 dport=58619 mark=0 use=1

  Bypassing Sec Groups firewalls entirely by disabling IPtable filtering
  on Linux bridges resolves the issue.

  net.bridge.bridge-nf-call-iptables=0

  Software Versions:
  Neutron: Kilo 2015.1.1
  Nova: Kilo 2015.1.1
  OS: CentOS 7.1

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1514017/+subscriptions