kernel-packages team mailing list archive

[Jay Vosburgh <1409123@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Virtualized network topologies that utilize encapsulation (e.g., VXLAN)
and bridging  may experience kernel errors of the format:

[ 4297.761899] eth0: hw csum failure
[ 4297.765210] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           OE  3.18.0-rc4
-nn+ #22
[ 4297.765212] Hardware name: LENOVO 0829F3U/To be filled by O.E.M., BIOS 90KT15
AUS 07/21/2010
[ 4297.765216]  0000000000000000 ffff88013fc03ba8 ffffffff8172f026 0000000000000
001
[ 4297.765219]  ffff88013870e000 ffff88013fc03bc8 ffffffff8162ba52 ffffffff8161c
1a0
[ 4297.765221]  ffff8800afdf1000 ffff88013fc03c08 ffffffff8162325c ffff88013870e
000
[ 4297.765223] Call Trace:
[ 4297.765224]  <IRQ>  [<ffffffff8172f026>] dump_stack+0x46/0x58
[ 4297.765235]  [<ffffffff8162ba52>] netdev_rx_csum_fault+0x42/0x50
[ 4297.765238]  [<ffffffff8161c1a0>] ? skb_push+0x40/0x40
[ 4297.765240]  [<ffffffff8162325c>] __skb_checksum_complete+0xbc/0xd0
[ 4297.765243]  [<ffffffff8168c602>] tcp_v4_rcv+0x2e2/0x950
[ 4297.765246]  [<ffffffff81666ca0>] ? ip_rcv_finish+0x360/0x360
[ 4297.765248]  [<ffffffff81660224>] ? nf_hook_slow+0x74/0x130
[ 4297.765250]  [<ffffffff81666ca0>] ? ip_rcv_finish+0x360/0x360
[ 4297.765253]  [<ffffffff81666d4c>] ip_local_deliver_finish+0xac/0x220
[ 4297.765255]  [<ffffffff81667058>] ip_local_deliver+0x48/0x80
[ 4297.765257]  [<ffffffff816669c1>] ip_rcv_finish+0x81/0x360
[ 4297.765259]  [<ffffffff81667332>] ip_rcv+0x2a2/0x3f0
[ 4297.765261]  [<ffffffff8162e932>] __netif_receive_skb_core+0x562/0x7a0
[ 4297.765263]  [<ffffffff8162eb88>] __netif_receive_skb+0x18/0x60
[ 4297.765265]  [<ffffffff8162f8f6>] process_backlog+0xa6/0x150

The backtrace may vary, stacks descending into conntrack have also been
observed:

Call Trace:
 <IRQ>  [<ffffffff8171a324>] dump_stack+0x45/0x56
 [<ffffffff8161bfba>] netdev_rx_csum_fault+0x3a/0x40
 [<ffffffff81614782>] __skb_checksum_complete_head+0x62/0x70
 [<ffffffff816147a1>] __skb_checksum_complete+0x11/0x20
 [<ffffffff816a3eac>] nf_ip_checksum+0xcc/0x100
 [<ffffffffa04df33b>] udp_error+0xdb/0x1f0 [nf_conntrack]
 [<ffffffffa04d926e>] nf_conntrack_in+0xee/0xb40 [nf_conntrack]
 [<ffffffffa0307653>] ? do_execute_actions+0x2e3/0xab0 [openvswitch]
 [<ffffffffa0307e4b>] ? ovs_execute_actions+0x2b/0x30 [openvswitch]
 [<ffffffff81654540>] ? inet_del_offload+0x40/0x40
 [<ffffffffa03b52e2>] ipv4_conntrack_in+0x22/0x30 [nf_conntrack_ipv4]
 [<ffffffff8164e0aa>] nf_iterate+0x9a/0xb0
 [<ffffffff81654540>] ? inet_del_offload+0x40/0x40
 [<ffffffff8164e134>] nf_hook_slow+0x74/0x130
 [<ffffffff81654540>] ? inet_del_offload+0x40/0x40
 [<ffffffff81654f68>] ip_rcv+0x2f8/0x3d0

The root cause of this is twofold:

First, the kernel handling of forwarded packets that have been
encapsulated (e.g., from VXLAN) for devices that support
CHECKSUM_COMPLETE checksum offload fails to update the running checksum
when decapsulating the packet.

Second, for the enic device itself, the hardware is not correctly
computing the checksum for some cases.

Both of these issues are patched in mainline:

commit 17e96834fd35997ca7cdfbf15413bcd5a36ad448 
Author: Govindarajulu Varadarajan <_govind@xxxxxxx> 
Date: Thu Dec 18 15:58:42 2014 +0530 

enic: fix rx skb checksum

commit 2c26d34bbcc0b3f30385d5587aa232289e2eed8e 
Author: Jay Vosburgh <jay.vosburgh@xxxxxxxxxxxxx> 
Date: Fri Dec 19 15:32:00 2014 -0800 

net/core: Handle csum for CHECKSUM_COMPLETE VXLAN forwarding

** Affects: linux (Ubuntu)
     Importance: Undecided
     Assignee: Jay Vosburgh (jvosburgh)
         Status: New

** Changed in: linux (Ubuntu)
     Assignee: (unassigned) => Jay Vosburgh (jvosburgh)

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1409123

Title:
  "hw csum failure" in encapsulated network topolgies

Status in linux package in Ubuntu:
  New

Bug description:
  Virtualized network topologies that utilize encapsulation (e.g.,
  VXLAN) and bridging  may experience kernel errors of the format:

  [ 4297.761899] eth0: hw csum failure
  [ 4297.765210] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           OE  3.18.0-rc4
  -nn+ #22
  [ 4297.765212] Hardware name: LENOVO 0829F3U/To be filled by O.E.M., BIOS 90KT15
  AUS 07/21/2010
  [ 4297.765216]  0000000000000000 ffff88013fc03ba8 ffffffff8172f026 0000000000000
  001
  [ 4297.765219]  ffff88013870e000 ffff88013fc03bc8 ffffffff8162ba52 ffffffff8161c
  1a0
  [ 4297.765221]  ffff8800afdf1000 ffff88013fc03c08 ffffffff8162325c ffff88013870e
  000
  [ 4297.765223] Call Trace:
  [ 4297.765224]  <IRQ>  [<ffffffff8172f026>] dump_stack+0x46/0x58
  [ 4297.765235]  [<ffffffff8162ba52>] netdev_rx_csum_fault+0x42/0x50
  [ 4297.765238]  [<ffffffff8161c1a0>] ? skb_push+0x40/0x40
  [ 4297.765240]  [<ffffffff8162325c>] __skb_checksum_complete+0xbc/0xd0
  [ 4297.765243]  [<ffffffff8168c602>] tcp_v4_rcv+0x2e2/0x950
  [ 4297.765246]  [<ffffffff81666ca0>] ? ip_rcv_finish+0x360/0x360
  [ 4297.765248]  [<ffffffff81660224>] ? nf_hook_slow+0x74/0x130
  [ 4297.765250]  [<ffffffff81666ca0>] ? ip_rcv_finish+0x360/0x360
  [ 4297.765253]  [<ffffffff81666d4c>] ip_local_deliver_finish+0xac/0x220
  [ 4297.765255]  [<ffffffff81667058>] ip_local_deliver+0x48/0x80
  [ 4297.765257]  [<ffffffff816669c1>] ip_rcv_finish+0x81/0x360
  [ 4297.765259]  [<ffffffff81667332>] ip_rcv+0x2a2/0x3f0
  [ 4297.765261]  [<ffffffff8162e932>] __netif_receive_skb_core+0x562/0x7a0
  [ 4297.765263]  [<ffffffff8162eb88>] __netif_receive_skb+0x18/0x60
  [ 4297.765265]  [<ffffffff8162f8f6>] process_backlog+0xa6/0x150

  The backtrace may vary, stacks descending into conntrack have also
  been observed:

  Call Trace:
   <IRQ>  [<ffffffff8171a324>] dump_stack+0x45/0x56
   [<ffffffff8161bfba>] netdev_rx_csum_fault+0x3a/0x40
   [<ffffffff81614782>] __skb_checksum_complete_head+0x62/0x70
   [<ffffffff816147a1>] __skb_checksum_complete+0x11/0x20
   [<ffffffff816a3eac>] nf_ip_checksum+0xcc/0x100
   [<ffffffffa04df33b>] udp_error+0xdb/0x1f0 [nf_conntrack]
   [<ffffffffa04d926e>] nf_conntrack_in+0xee/0xb40 [nf_conntrack]
   [<ffffffffa0307653>] ? do_execute_actions+0x2e3/0xab0 [openvswitch]
   [<ffffffffa0307e4b>] ? ovs_execute_actions+0x2b/0x30 [openvswitch]
   [<ffffffff81654540>] ? inet_del_offload+0x40/0x40
   [<ffffffffa03b52e2>] ipv4_conntrack_in+0x22/0x30 [nf_conntrack_ipv4]
   [<ffffffff8164e0aa>] nf_iterate+0x9a/0xb0
   [<ffffffff81654540>] ? inet_del_offload+0x40/0x40
   [<ffffffff8164e134>] nf_hook_slow+0x74/0x130
   [<ffffffff81654540>] ? inet_del_offload+0x40/0x40
   [<ffffffff81654f68>] ip_rcv+0x2f8/0x3d0

  The root cause of this is twofold:

  First, the kernel handling of forwarded packets that have been
  encapsulated (e.g., from VXLAN) for devices that support
  CHECKSUM_COMPLETE checksum offload fails to update the running
  checksum when decapsulating the packet.

  Second, for the enic device itself, the hardware is not correctly
  computing the checksum for some cases.

  Both of these issues are patched in mainline:

  commit 17e96834fd35997ca7cdfbf15413bcd5a36ad448 
  Author: Govindarajulu Varadarajan <_govind@xxxxxxx> 
  Date: Thu Dec 18 15:58:42 2014 +0530 

  enic: fix rx skb checksum

  commit 2c26d34bbcc0b3f30385d5587aa232289e2eed8e 
  Author: Jay Vosburgh <jay.vosburgh@xxxxxxxxxxxxx> 
  Date: Fri Dec 19 15:32:00 2014 -0800 

  net/core: Handle csum for CHECKSUM_COMPLETE VXLAN forwarding

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1409123/+subscriptions