yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2046196] Re: Unicast ICMPv6 Router Advertisement packets to VM's dropped by OVS firewall driver

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: yatin <2046196@xxxxxxxxxxxxxxxxxx>
Date: Mon, 18 Dec 2023 05:17:56 -0000
Reply-to: Bug 2046196 <2046196@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
*** This bug is a duplicate of bug 1958643 ***
    https://bugs.launchpad.net/bugs/1958643

Thanks @Stanislav for the confirmation, will close it as Duplicate.

** This bug has been marked a duplicate of bug 1958643
   Unicast RA messages for a VM are filtered out by ovs rules

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2046196

Title:
  Unicast ICMPv6 Router Advertisement packets to VM's dropped by OVS
  firewall driver

Status in neutron:
  Incomplete

Bug description:
  When use Open vSwitch Firewall Driver we noticed that ICMPv6 RA
  unicast packets are not reaching the VM.

  
  =========================================Troubleshooting flow:===========================================

  1) Catch ICMPv6 RA package on physical hypervisor bonding interface:

  #tcpdump -XXvpnei bond0 -Q in -c 1 "icmp6[0] = 134 and ether host fa:16:3e:68:e8:19"
  dropped privs to tcpdump
  tcpdump: listening on bond0, link-type EN10MB (Ethernet), capture size 262144 bytes
  21:56:43.669691 26:28:b0:96:c0:c7 > fa:16:3e:68:e8:19, ethertype 802.1Q (0x8100), length 162: vlan 3254, p 0, ethertype IPv6, (flowlabel 0x7cc6b, hlim 255, next-header ICMPv6 (58) payload length: 104) fe80::2428:b0ff:fe96:c0c7 > fe80::f816:3eff:fe68:e819: [icmp6 sum ok] ICMP6, router advertisement, length 104
  	hop limit 64, Flags [managed], pref medium, router lifetime 6s, reachable time 0ms, retrans timer 0ms
  	  prefix info option (3), length 32 (4): 2a05:fc1:200::/64, Flags [onlink, auto], valid time 2592000s, pref. time 14400s
  	  rdnss option (25), length 40 (5):  lifetime 2s, addr: 2a05:fc1::2 addr: 2a05:fc1::3
  	  source link-address option (1), length 8 (1): 26:28:b0:96:c0:c7
  	  advertisement interval option (7), length 8 (1):  2000ms
  	0x0000:  fa16 3e68 e819 2628 b096 c0c7 8100 0cb6  ..>h..&(........
  	0x0010:  86dd 6007 cc6b 0068 3aff fe80 0000 0000  ..`..k.h:.......
  	0x0020:  0000 2428 b0ff fe96 c0c7 fe80 0000 0000  ..$(............
  	0x0030:  0000 f816 3eff fe68 e819 8600 10d2 4080  ....>..h......@.
  	0x0040:  0006 0000 0000 0000 0000 0304 40c0 0027  ............@..'
  	0x0050:  8d00 0000 3840 0000 0000 2a05 0fc1 0200  ....8@....*.....
  	0x0060:  0000 0000 0000 0000 0000 1905 0000 0000  ................
  	0x0070:  0002 2a05 0fc1 0000 0000 0000 0000 0000  ..*.............
  	0x0080:  0002 2a05 0fc1 0000 0000 0000 0000 0000  ..*.............
  	0x0090:  0003 0101 2628 b096 c0c7 0701 0000 0000  ....&(..........
  	0x00a0:  07d0                                     ..

  2) Trace the package using "ofproto/trace":

  #ovs-appctl ofproto/trace br-int in_port=1 fa163e68e8192628b096c0c781000cb686dd6007cc6b00683afffe800000000000002428b0fffe96c0c7fe80000000000000f8163efffe68e819860010d2408000060000000000000000030440c000278d0000003840000000002a050fc102000000000000000000000019050000000000022a050fc10000000000000000000000022a050fc100000000000000000000000301012628b096c0c707010000000007d0
  Flow: icmp6,in_port=1,dl_vlan=3254,dl_vlan_pcp=0,vlan_tci1=0x0000,dl_src=26:28:b0:96:c0:c7,dl_dst=fa:16:3e:68:e8:19,ipv6_src=fe80::2428:b0ff:fe96:c0c7,ipv6_dst=fe80::f816:3eff:fe68:e819,ipv6_label=0x7cc6b,nw_tos=0,nw_ecn=0,nw_ttl=255,icmp_type=134,icmp_code=0

  bridge("br-int")
  ----------------
   0. in_port=1,dl_vlan=3254, priority 3, cookie 0x4ef408376e507615
      set_field:4097->vlan_vid
      goto_table:60
  60. dl_vlan=1,dl_dst=fa:16:3e:68:e8:19, priority 90, cookie 0x4ef408376e507615
      set_field:0x9e->reg5
      set_field:0x1->reg6
      pop_vlan
      resubmit(,81)
  81. ct_state=-trk,ipv6,reg5=0x9e, priority 90, cookie 0x4ef408376e507615
      ct(table=82,zone=NXM_NX_REG6[0..15])
      drop
       -> A clone of the packet is forked to recirculate. The forked pipeline will be resumed at table 82.
       -> Sets the packet to an untracked state, and clears all the conntrack fields.

  Final flow: icmp6,reg5=0x9e,reg6=0x1,in_port=1,vlan_tci=0x0000,dl_src=26:28:b0:96:c0:c7,dl_dst=fa:16:3e:68:e8:19,ipv6_src=fe80::2428:b0ff:fe96:c0c7,ipv6_dst=fe80::f816:3eff:fe68:e819,ipv6_label=0x7cc6b,nw_tos=0,nw_ecn=0,nw_ttl=255,icmp_type=134,icmp_code=0
  Megaflow: recirc_id=0,ct_state=-trk,eth,icmp6,in_port=1,dl_vlan=3254,dl_vlan_pcp=0,dl_dst=fa:16:3e:68:e8:19,nw_frag=no,icmp_type=0x86/0xff
  Datapath actions: pop_vlan,ct(zone=1),recirc(0xc30495)

  ===============================================================================
  recirc(0xc30495) - resume conntrack with default ct_state=trk|new (use --ct-next to customize)
  ===============================================================================

  Flow:
  recirc_id=0xc30495,ct_state=new|trk,ct_zone=1,eth,icmp6,reg5=0x9e,reg6=0x1,in_port=1,vlan_tci=0x0000,dl_src=26:28:b0:96:c0:c7,dl_dst=fa:16:3e:68:e8:19,ipv6_src=fe80::2428:b0ff:fe96:c0c7,ipv6_dst=fe80::f816:3eff:fe68:e819,ipv6_label=0x7cc6b,nw_tos=0,nw_ecn=0,nw_ttl=255,icmp_type=134,icmp_code=0

  bridge("br-int")
  ----------------
      thaw
          Resuming from table 82
  82. ct_state=+new-est,ipv6,reg5=0x9e, priority 74, cookie 0x4ef408376e507615
      ct(commit,zone=NXM_NX_REG6[0..15])
      drop
       -> Sets the packet to an untracked state, and clears all the conntrack fields.
      output:158
      resubmit(,92)
  92. priority 0, cookie 0x4ef408376e507615
      drop

  Final flow: recirc_id=0xc30495,eth,icmp6,reg5=0x9e,reg6=0x1,in_port=1,vlan_tci=0x0000,dl_src=26:28:b0:96:c0:c7,dl_dst=fa:16:3e:68:e8:19,ipv6_src=fe80::2428:b0ff:fe96:c0c7,ipv6_dst=fe80::f816:3eff:fe68:e819,ipv6_label=0x7cc6b,nw_tos=0,nw_ecn=0,nw_ttl=255,icmp_type=134,icmp_code=0
  Megaflow: recirc_id=0xc30495,ct_state=+new-est-rel-rpl,eth,ipv6,in_port=1,nw_frag=no
  Datapath actions: ct(commit,zone=1),125

  3) We've verified that it's all about OVS and the packet is discarded
  by the rule.


  ============================================Fixing the
  issue:============================================

  The documentation for the OVS Firewall Driver (https://docs.openstack.org/neutron/latest/contributor/internals/openvswitch_firewall.html#rules-example-with-explanation) lists the rule tables for neighbor solicitation and neighbor advertisement, but there is nothing about router advertisement. We have studied the source code that creates the necessary rules and paid attention to the ICMPV6_ALLOWED_INGRESS_TYPES variable that accepts the list of ICMPv6 types.
  We just added the required type there and it solved the problem.

  neutron/agent/firewall.py
  ICMPV6_ALLOWED_INGRESS_TYPES = (n_const.ICMPV6_TYPE_MLD_QUERY,
  +                               n_const.ICMPV6_TYPE_RA,
                                  n_const.ICMPV6_TYPE_NS,
                                  n_const.ICMPV6_TYPE_NA)

  It solved the issue.

  We got successful trace:
  # docker exec openvswitch_vswitchd ovs-appctl ofproto/trace br-ext in_port=1 fa163e68e8192628b096c0c781000cb686dd6007cc6b00683afffe800000000000002428b0fffe96c0c7fe80000000000000f8163efffe68e819860010d2408000060000000000000000030440c000278d0000003840000000002a050fc102000000000000000000000019050000000000022a050fc10000000000000000000000022a050fc100000000000000000000000301012628b096c0c707010000000007d0
  Flow: icmp6,in_port=1,dl_vlan=3254,dl_vlan_pcp=0,vlan_tci1=0x0000,dl_src=26:28:b0:96:c0:c7,dl_dst=fa:16:3e:68:e8:19,ipv6_src=fe80::2428:b0ff:fe96:c0c7,ipv6_dst=fe80::f816:3eff:fe68:e819,ipv6_label=0x7cc6b,nw_tos=0,nw_ecn=0,nw_ttl=255,icmp_type=134,icmp_code=0

  bridge("br-ext")
  ----------------
   0. priority 0, cookie 0x919d5b8cf75468f6
      NORMAL
       -> forwarding to learned port

  bridge("br-int")
  ----------------
   0. in_port=1,dl_vlan=3254, priority 3, cookie 0x99db92e60316827
      set_field:4097->vlan_vid
      goto_table:60
  60. dl_vlan=1,dl_dst=fa:16:3e:68:e8:19, priority 90, cookie 0x99db92e60316827
      set_field:0x9e->reg5
      set_field:0x1->reg6
      pop_vlan
      resubmit(,81)
  81. icmp6,reg5=0x9e,icmp_type=134, priority 100, cookie 0x99db92e60316827
      output:158

  Final flow: unchanged
  Megaflow: recirc_id=0,eth,icmp6,in_port=1,dl_vlan=3254,dl_vlan_pcp=0,dl_src=26:28:b0:96:c0:c7,dl_dst=fa:16:3e:68:e8:19,nw_frag=no,icmp_type=0x86/0xff
  Datapath actions: pop_vlan,125

  
  ============================================Affected versions:===========================================

  We made tests:
  - Ussuri 
  - Zed
  - Master branch

  
  ===============================================Conclusions:==============================================

  1) I think that correct work has been broken by this one commit:
  https://opendev.org/openstack/neutron/commit/0dcf3d20c2e5c2592e9674e7277acce4eff98341

  2) I suggest returning n_const.ICMPV6_TYPE_RA to the
  ICMPV6_ALLOWED_INGRESS_TYPES variable and eliminate the duplicated
  ICMPv6 RA rule from the iptables firewall driver.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2046196/+subscriptions
References

[Bug 2046196] [NEW] Unicast ICMPv6 Router Advertisement packets to VM's dropped by OVS firewall driver
From: Stanislav Tretiakov, 2023-12-11