yahoo-eng-team team mailing list archive

[Carl Baldwin <carl@xxxxxxxxxxxxx>]

At this point, given that fwaas is totally up in the air, I don't think
we're going to take on anymore DVR / FWaaS bugs.

** Changed in: neutron
       Status: Confirmed => Won't Fix

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1513574

Title:
  firewall rules on DVR FIP fails to work for ingress traffic

Status in neutron:
  Won't Fix

Bug description:
  =====
  my env
  =====
  controller +network node(dvr_snat) + 2 compute nodes(dvr)
  DVR: enable DVR when using devstack to deploy this env
  FWaaS: manually git clone neutron-fwaas and to configure, using iptables as driver

  
  ====
  steps
  ====
  1) create net, subnet, boot VM-1 on CN-1, VM-2 on CN-2, create router, and attach subnet onto router.
  2) create external network, set as router gateway net, create 2 floating IPs and associate to two VMs.
  3) confirm DVR FIP works: fip ns created, iptable rules updated in qrouter ns, two VMs are pingable by floating IP.
  floating IP like: 192.168.0.4 and 192.168.0.5
  4) create firewall rules, firewall policy and create firewall on router. 
  firewall rule like: 
      fw-r1: ICMP, source: 192.168.0.184/29(none), dest: 192.168.0.0/28(none), allow
      fw-r2: ICMP, source: 192.168.0.0/28(none), dest: 192.168.0.184/29(none), allow
  5) confirm firewall rules updated in qrouter ns.
  6) on host who has IP like 192.168.0.190, try to ping floating IPs mentioned in step 3.
  expected: floating IPs should be pingable (for IP 192.168.0.190 is in 192.168.0.184/29, and two firewall rules allows)
  observed: no response, "100% packet loss" from ping command. floating IP fail to ping.

  
  ========
  more details
  ========
  ----------------------------
  firewall iptable rules:
  ----------------------------
  -A INPUT -j neutron-l3-agent-INPUT
  -A FORWARD -j neutron-filter-top
  -A FORWARD -j neutron-l3-agent-FORWARD
  -A OUTPUT -j neutron-filter-top
  -A OUTPUT -j neutron-l3-agent-OUTPUT
  -A neutron-filter-top -j neutron-l3-agent-local
  -A neutron-l3-agent-FORWARD -o rfp-+ -j neutron-l3-agent-iv4322a9b15
  -A neutron-l3-agent-FORWARD -i rfp-+ -j neutron-l3-agent-ov4322a9b15
  -A neutron-l3-agent-FORWARD -o rfp-+ -j neutron-l3-agent-fwaas-defau
  -A neutron-l3-agent-FORWARD -i rfp-+ -j neutron-l3-agent-fwaas-defau
  -A neutron-l3-agent-INPUT -m mark --mark 0x1/0xffff -j ACCEPT
  -A neutron-l3-agent-INPUT -p tcp -m tcp --dport 9697 -j DROP
  -A neutron-l3-agent-fwaas-defau -j DROP
  -A neutron-l3-agent-iv4322a9b15 -m state --state INVALID -j DROP
  -A neutron-l3-agent-iv4322a9b15 -m state --state RELATED,ESTABLISHED -j ACCEPT
  -A neutron-l3-agent-iv4322a9b15 -s 192.168.0.0/28 -d 192.168.0.184/29 -p icmp -j ACCEPT
  -A neutron-l3-agent-iv4322a9b15 -s 192.168.0.184/29 -d 192.168.0.0/28 -p icmp -j ACCEPT
  -A neutron-l3-agent-ov4322a9b15 -m state --state INVALID -j DROP
  -A neutron-l3-agent-ov4322a9b15 -m state --state RELATED,ESTABLISHED -j ACCEPT
  -A neutron-l3-agent-ov4322a9b15 -s 192.168.0.0/28 -d 192.168.0.184/29 -p icmp -j ACCEPT
  -A neutron-l3-agent-ov4322a9b15 -s 192.168.0.184/29 -d 192.168.0.0/28 -p icmp -j ACCEPT

  -----------------------------------
  DVR FIP nat iptable rules:
  -----------------------------------
  1) for 192.168.0.4:
  -A PREROUTING -j neutron-l3-agent-PREROUTING
  -A OUTPUT -j neutron-l3-agent-OUTPUT
  -A POSTROUTING -j neutron-l3-agent-POSTROUTING
  -A POSTROUTING -j neutron-postrouting-bottom
  -A neutron-l3-agent-OUTPUT -d 192.168.0.4/32 -j DNAT --to-destination 20.0.1.7
  -A neutron-l3-agent-POSTROUTING ! -i rfp-4bf3186c-d ! -o rfp-4bf3186c-d -m conntrack ! --ctstate DNAT -j ACCEPT
  -A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
  -A neutron-l3-agent-PREROUTING -d 192.168.0.4/32 -j DNAT --to-destination 20.0.1.7
  -A neutron-l3-agent-float-snat -s 20.0.1.7/32 -j SNAT --to-source 192.168.0.4
  -A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
  -A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat

  2) for 192.168.0.5:
  -A PREROUTING -j neutron-l3-agent-PREROUTING
  -A OUTPUT -j neutron-l3-agent-OUTPUT
  -A POSTROUTING -j neutron-l3-agent-POSTROUTING
  -A POSTROUTING -j neutron-postrouting-bottom
  -A neutron-l3-agent-OUTPUT -d 192.168.0.5/32 -j DNAT --to-destination 20.0.1.6
  -A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
  -A neutron-l3-agent-PREROUTING -d 192.168.0.5/32 ! -i qr-+ -j DNAT --to-destination 20.0.1.6
  -A neutron-l3-agent-float-snat -s 20.0.1.6/32 -j SNAT --to-source 192.168.0.5
  -A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
  -A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat

  ----------------------------------------------------------------------
  tcpdump result: (192.168.0.190 ping 192.168.0.4)
  ----------------------------------------------------------------------
  1) on fg in fip ns, ingress traffic caught:
  fa:16:3e:b3:3e:8c > fa:16:3e:9d:ea:ed, ethertype IPv4 (0x0800), length 98: 192.168.0.190 > 192.168.0.4: ICMP echo request, id 28356, seq 31, length 64
  and fg:
  40: fg-59c9ce49-3a: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default 
      link/ether fa:16:3e:9d:ea:ed brd ff:ff:ff:ff:ff:ff
      inet 192.168.0.133/24 brd 192.168.0.255 scope global fg-59c9ce49-3a
         valid_lft forever preferred_lft forever
      inet6 fe80::f816:3eff:fe9d:eaed/64 scope link 
         valid_lft forever preferred_lft forever

  2) on fpr in fip ns, ingress traffic caught:
  a2:c4:79:bf:c0:1a > aa:ab:39:2f:ac:df, ethertype IPv4 (0x0800), length 98: 192.168.0.190 > 192.168.0.4: ICMP echo request, id 28356, seq 159, length 64
  and fpr:
  13: fpr-4bf3186c-d: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
      link/ether a2:c4:79:bf:c0:1a brd ff:ff:ff:ff:ff:ff
      inet 169.254.31.143/31 scope global fpr-4bf3186c-d
         valid_lft forever preferred_lft forever
      inet6 fe80::a0c4:79ff:febf:c01a/64 scope link 
         valid_lft forever preferred_lft forever
  and rfp:
  10: rfp-4bf3186c-d: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
      link/ether aa:ab:39:2f:ac:df brd ff:ff:ff:ff:ff:ff
      inet 169.254.31.142/31 scope global rfp-4bf3186c-d
         valid_lft forever preferred_lft forever
      inet 192.168.0.4/32 brd 192.168.0.4 scope global rfp-4bf3186c-d
         valid_lft forever preferred_lft forever
      inet6 fe80::a8ab:39ff:fe2f:acdf/64 scope link 
         valid_lft forever preferred_lft forever

  3) on rfp in qrouter ns, ingress traffic caught:
  a2:c4:79:bf:c0:1a > aa:ab:39:2f:ac:df, ethertype IPv4 (0x0800), length 98: 192.168.0.190 > 192.168.0.4: ICMP echo request, id 28356, seq 372, length 64

  4) on qr in qrouter ns, no traffic caught  :(

  
  ========
  idea/discuss
  ========
  in my understanding, we have DNAT rules in PREROUTING chain, like:
  -A neutron-l3-agent-PREROUTING -d 192.168.0.4/32 -j DNAT --to-destination 20.0.1.7
  will change ingress traffic destination IP from 192.168.0.4(firewall know and allow) to 20.0.1.7(firewall dont know and deny by neutron-l3-agent-fwaas-defau).
  So before firewall rules in FORWARD related chains like:
  -A neutron-l3-agent-iv4322a9b15 -s 192.168.0.0/28 -d 192.168.0.184/29 -p icmp -j ACCEPT
  -A neutron-l3-agent-iv4322a9b15 -s 192.168.0.184/29 -d 192.168.0.0/28 -p icmp -j ACCEPT
  have chance to accept, the ingress traffic has already been changed, and will go to neutron-l3-agent-fwaas-defau chain and be dropped.

  an idea is try to move firewall rules into fip ns(but this will raise
  other potential problems, for firewall rules will no longer be
  isolated by namespace), I tested this, it can work.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1513574/+subscriptions