yahoo-eng-team team mailing list archive

[Michal Nasiadka <1851194@xxxxxxxxxxxxxxxxxx>]

bug report is for Stein and Stein is EOL - please test out on newer
version.

** Changed in: kolla-ansible
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1851194

Title:
  FWaaSv2 configures iptables with invalid port name

Status in kolla-ansible:
  Invalid
Status in neutron:
  New

Bug description:
  This might be a duplicate of:
  https://bugs.launchpad.net/neutron/+bug/1798577

  however, I see the same issue with floating IPs and DVR, not just
  SNAT.  This breaks the FWaaSv2 service, resulting in no filtering.

  Deployment uses Kolla Ansible 8.0.1 (Stein) on CentOS, which installs neutron-fwaas from here:
  http://tarballs.openstack.org/neutron-fwaas/neutron-fwaas-15.0.0.tar.gz

  DVR is being used, but the issue appears to exist on both distributed
  routers and centralized routers on the network nodes (both qrouter and
  snat namespaces).

  I am attaching a small bash script that creates a firewall rule,
  policy, and group to block TCP port 25 outbound.  When this didn't
  work (outbound SMTP traffic was not blocked), I looked at the iptables
  rules on the network node and compute nodes.

  
  These are the interfaces in the qrouter namespace of a test router that has the issue with fwaasv2:

  2: rfp-3f6273be-2@if19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
      link/ether be:8b:2c:40:dc:e5 brd ff:ff:ff:ff:ff:ff link-netnsid 0
      inet 169.254.125.2/31 scope global rfp-3f6273be-2
         valid_lft forever preferred_lft forever
      inet6 fe80::bc8b:2cff:fe40:dce5/64 scope link
         valid_lft forever preferred_lft forever

  1118: qr-a418f15b-fa: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
      link/ether fa:16:3e:6f:fe:2c brd ff:ff:ff:ff:ff:ff
      inet 192.168.99.254/24 brd 192.168.99.255 scope global qr-a418f15b-fa
         valid_lft forever preferred_lft forever
      inet6 fe80::f816:3eff:fe6f:fe2c/64 scope link
         valid_lft forever preferred_lft forever

  The is the iptables config (iptables -S), with some notes injected:

  -P INPUT ACCEPT
  -P FORWARD ACCEPT
  -P OUTPUT ACCEPT
  -N neutron-filter-top
  -N neutron-l3-agent-FORWARD
  -N neutron-l3-agent-INPUT
  -N neutron-l3-agent-OUTPUT
  -N neutron-l3-agent-accepted
  -N neutron-l3-agent-dropped
  -N neutron-l3-agent-fwaas-defau
  -N neutron-l3-agent-iv4ff1b0489
  -N neutron-l3-agent-local
  -N neutron-l3-agent-ov4ff1b0489
  -N neutron-l3-agent-rejected
  -N neutron-l3-agent-scope
  -A INPUT -j neutron-l3-agent-INPUT
  -A FORWARD -j neutron-filter-top
  -A FORWARD -j neutron-l3-agent-FORWARD
  -A OUTPUT -j neutron-filter-top
  -A OUTPUT -j neutron-l3-agent-OUTPUT
  -A neutron-filter-top -j neutron-l3-agent-local
  -A neutron-l3-agent-FORWARD -j neutron-l3-agent-scope

  # NOTE: These 4 rules have the wrong interface - note that "a418f15b-f" is
  # the name of the qr interface, NOT the rfp interface (see above interface list):
  -A neutron-l3-agent-FORWARD -o rfp-a418f15b-f -j neutron-l3-agent-iv4ff1b0489
  -A neutron-l3-agent-FORWARD -i rfp-a418f15b-f -j neutron-l3-agent-ov4ff1b0489
  -A neutron-l3-agent-FORWARD -o rfp-a418f15b-f -j neutron-l3-agent-fwaas-defau
  -A neutron-l3-agent-FORWARD -i rfp-a418f15b-f -j neutron-l3-agent-fwaas-defau

  -A neutron-l3-agent-INPUT -m mark --mark 0x1/0xffff -j ACCEPT
  -A neutron-l3-agent-INPUT -p tcp -m tcp --dport 9697 -j DROP
  -A neutron-l3-agent-accepted -j ACCEPT
  -A neutron-l3-agent-dropped -j DROP
  -A neutron-l3-agent-fwaas-defau -j neutron-l3-agent-dropped
  -A neutron-l3-agent-iv4ff1b0489 -m state --state INVALID -j neutron-l3-agent-dropped
  -A neutron-l3-agent-iv4ff1b0489 -m state --state RELATED,ESTABLISHED -j ACCEPT
  -A neutron-l3-agent-iv4ff1b0489 -p tcp -m tcp --dport 25 -j neutron-l3-agent-dropped
  -A neutron-l3-agent-ov4ff1b0489 -m state --state INVALID -j neutron-l3-agent-dropped
  -A neutron-l3-agent-ov4ff1b0489 -m state --state RELATED,ESTABLISHED -j ACCEPT
  -A neutron-l3-agent-ov4ff1b0489 -p tcp -m tcp --dport 25 -j neutron-l3-agent-dropped
  -A neutron-l3-agent-rejected -j REJECT --reject-with icmp-port-unreachable

  # NOTE: These two rules use the correct interface names:
  -A neutron-l3-agent-scope -o rfp-3f6273be-2 -m mark ! --mark 0x4000000/0xffff0000 -j DROP
  -A neutron-l3-agent-scope -o qr-a418f15b-fa -m mark ! --mark 0x4000000/0xffff0000 -j DROP

  
  This obviously breaks the FWaaSv2 pretty severely (iptables simply ignores the rules with the incorrect interface names).

  
  The same issue occurs in the SNAT namespace...

  The SNAT namespace interfaces:

  1115: ha-8632c48f-29: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9150 qdisc noqueue state UNKNOWN group default qlen 1000
      link/ether fa:16:3e:ee:dd:91 brd ff:ff:ff:ff:ff:ff
      inet 169.254.192.1/18 brd 169.254.255.255 scope global ha-8632c48f-29
         valid_lft forever preferred_lft forever
      inet6 fe80::f816:3eff:feee:dd91/64 scope link
         valid_lft forever preferred_lft forever

  1143: sg-8f9c085a-be: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
      link/ether fa:16:3e:3a:31:f8 brd ff:ff:ff:ff:ff:ff

  1155: qg-8e38e9f5-28: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
      link/ether fa:16:3e:f9:3a:d3 brd ff:ff:ff:ff:ff:ff

  
  The iptables (iptables -S) output - with notes injected:

  -P INPUT ACCEPT
  -P FORWARD ACCEPT
  -P OUTPUT ACCEPT
  -N neutron-filter-top
  -N neutron-l3-agent-FORWARD
  -N neutron-l3-agent-INPUT
  -N neutron-l3-agent-OUTPUT
  -N neutron-l3-agent-accepted
  -N neutron-l3-agent-dropped
  -N neutron-l3-agent-fwaas-defau
  -N neutron-l3-agent-iv4ff1b0489
  -N neutron-l3-agent-local
  -N neutron-l3-agent-ov4ff1b0489
  -N neutron-l3-agent-rejected
  -N neutron-l3-agent-scope
  -A INPUT -j neutron-l3-agent-INPUT
  -A FORWARD -j neutron-filter-top
  -A FORWARD -j neutron-l3-agent-FORWARD
  -A OUTPUT -j neutron-filter-top
  -A OUTPUT -j neutron-l3-agent-OUTPUT
  -A neutron-filter-top -j neutron-l3-agent-local
  -A neutron-l3-agent-FORWARD -j neutron-l3-agent-scope

  # NOTE: These 4 rules have the wrong interface - note that "a418f15b-fa" is
  # the name of the qr interface, NOT the sg interface.  It is actually the "rfp"
  # interface from the qrouter namespace!
  -A neutron-l3-agent-FORWARD -o sg-a418f15b-fa -j neutron-l3-agent-iv4ff1b0489
  -A neutron-l3-agent-FORWARD -i sg-a418f15b-fa -j neutron-l3-agent-ov4ff1b0489
  -A neutron-l3-agent-FORWARD -o sg-a418f15b-fa -j neutron-l3-agent-fwaas-defau
  -A neutron-l3-agent-FORWARD -i sg-a418f15b-fa -j neutron-l3-agent-fwaas-defau
  -A neutron-l3-agent-accepted -j ACCEPT
  -A neutron-l3-agent-dropped -j DROP
  -A neutron-l3-agent-fwaas-defau -j neutron-l3-agent-dropped
  -A neutron-l3-agent-iv4ff1b0489 -m state --state INVALID -j neutron-l3-agent-dropped
  -A neutron-l3-agent-iv4ff1b0489 -m state --state RELATED,ESTABLISHED -j ACCEPT
  -A neutron-l3-agent-iv4ff1b0489 -p tcp -m tcp --dport 25 -j neutron-l3-agent-dropped
  -A neutron-l3-agent-ov4ff1b0489 -m state --state INVALID -j neutron-l3-agent-dropped
  -A neutron-l3-agent-ov4ff1b0489 -m state --state RELATED,ESTABLISHED -j ACCEPT
  -A neutron-l3-agent-ov4ff1b0489 -p tcp -m tcp --dport 25 -j neutron-l3-agent-dropped
  -A neutron-l3-agent-rejected -j REJECT --reject-with icmp-port-unreachable

  # NOTE: This rule has the correct interface name:
  -A neutron-l3-agent-scope -o sg-8f9c085a-be -m mark ! --mark 0x4000000/0xffff0000 -j DROP

  
  FWaaSv2 is a pretty critical component, and I haven't found a patch that has been made after Stein (maybe I just didn't find it though).

  Eric

To manage notifications about this bug go to:
https://bugs.launchpad.net/kolla-ansible/+bug/1851194/+subscriptions