yahoo-eng-team team mailing list archive

[Oisin <oisin.omalley@xxxxxxxxxxxxx>]

Public bug reported:


Sec Group rules are set to allow all TCP connections in both directions.

When 2 instance on the same host but in different networks try to start
a TCP session, the TCP handshake never completes as the as the ACK
packet gets dropped as invalid by the security group iptable rules on
the Linux bridge. The SYN and SYN_ACK packets get through. This happens
on all libvirt hosts, when the 2 instances are the same host. When the
instance are on separate hosts, the ACK packet gets through and the
session is established.

VM1 --> SYN --> VM2
VM1 <-- SYN_ACK <-- VM2
VM1 --> ACK --> ???

In the case of the above running tcpdump on the host on the VM1 tap
interface shows the ACK packet entering the bridge, a tcpdump on qvb
interface on the far side of the bridge show the packet missing.


After examining iptable rule counters while attempting to open a TCP session, the following rule in the neutron-openvswi-od080d3d6-c chain is dropping the packets.

DROP       all  --  anywhere             anywhere             state
INVALID /* Drop packets that appear related to an existing connection
(e.g. TCP ACK/FIN) but do not have an entry in conntrack. */


conntrack shows the TCP session as being in the correct SYN_RECV state.

conntrack -L -s 10.0.116.15 -d 10.0.115.11 -p tcp
tcp      6 57 SYN_RECV src=10.0.116.15 dst=10.0.115.11 sport=58619 dport=22 src=10.0.115.11 dst=10.0.116.15 sport=22 dport=58619 mark=0 use=1


Bypassing Sec Groups firewalls entirely by disabling IPtable filtering on Linux bridges resolves the issue. 

net.bridge.bridge-nf-call-iptables=0

Software Versions:
Neutron: Kilo 2015.1.1
Nova: Kilo 2015.1.1
OS: CentOS 7.1

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1514017

Title:
  Sec Group IPtables dropping all TCP ACK as Invalid for all connections
  bettween instances on different networks on same libvirt host.

Status in neutron:
  New

Bug description:
  
  Sec Group rules are set to allow all TCP connections in both directions.

  When 2 instance on the same host but in different networks try to
  start a TCP session, the TCP handshake never completes as the as the
  ACK packet gets dropped as invalid by the security group iptable rules
  on the Linux bridge. The SYN and SYN_ACK packets get through. This
  happens on all libvirt hosts, when the 2 instances are the same host.
  When the instance are on separate hosts, the ACK packet gets through
  and the session is established.

  VM1 --> SYN --> VM2
  VM1 <-- SYN_ACK <-- VM2
  VM1 --> ACK --> ???

  In the case of the above running tcpdump on the host on the VM1 tap
  interface shows the ACK packet entering the bridge, a tcpdump on qvb
  interface on the far side of the bridge show the packet missing.

  
  After examining iptable rule counters while attempting to open a TCP session, the following rule in the neutron-openvswi-od080d3d6-c chain is dropping the packets.

  DROP       all  --  anywhere             anywhere             state
  INVALID /* Drop packets that appear related to an existing connection
  (e.g. TCP ACK/FIN) but do not have an entry in conntrack. */

  
  conntrack shows the TCP session as being in the correct SYN_RECV state.

  conntrack -L -s 10.0.116.15 -d 10.0.115.11 -p tcp
  tcp      6 57 SYN_RECV src=10.0.116.15 dst=10.0.115.11 sport=58619 dport=22 src=10.0.115.11 dst=10.0.116.15 sport=22 dport=58619 mark=0 use=1

  
  Bypassing Sec Groups firewalls entirely by disabling IPtable filtering on Linux bridges resolves the issue. 

  net.bridge.bridge-nf-call-iptables=0

  Software Versions:
  Neutron: Kilo 2015.1.1
  Nova: Kilo 2015.1.1
  OS: CentOS 7.1

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1514017/+subscriptions