kernel-packages team mailing list archive

Thread
Date
[Bug 1331219] Re: Kernel Bug at skb_segment + 0x95a

To: kernel-packages@xxxxxxxxxxxxxxxxxxx
From: Dave Chiluk <1331219@xxxxxxxxxxxxxxxxxx>
Date: Fri, 25 Jul 2014 19:50:16 -0000
Reply-to: Bug 1331219 <1331219@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Description changed:

+ SRU Justification:
+ Impact: A crash occurs with VXLAN packets when using GSO capable hardware such as the case with openstack neutron.
+ 
+ Fix: Upstream fix 5882a07c7 fixes a locking error that previously allowed
+ certain timings to hit this bug.
+ 
+ Testcase: Create and run a Neutron gateway with VXLANs, and stress the
+ network for an extended period of time.
+ 
+ 
  Kernel Bug affects 3.11-3.15 on machines using gso offload.
  
  Stack trace
  [ 871.025524] ------------[ cut here ]------------
  [ 871.030742] kernel BUG at /build/buildd/linux-3.13.0/net/core/skbuff.c:2903!
  [ 871.038669] invalid opcode: 0000 [#1] SMP
  [ 871.043318] Modules linked in: xt_nat xt_REDIRECT dccp_diag dccp tcp_diag udp_diag inet_diag unix_diag ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conn
  track nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp bridge ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables veth openvswitch gre vxlan ip_tunnel
   dm_crypt 8021q garp stp mrp llc bonding nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache sb_edac gpio_ich joydev edac_core x86_pkg_temp_thermal intel_powerclamp coretemp lpc_ich acpi_power_m
  eter kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd ipmi_si mac_hid lp parport btrfs xor raid6_pq libcrc32c
  hid_generic qla2xxx fnic libfcoe megaraid_sas
  [ 871.124025] igb usbhid libfc hid dca scsi_transport_fc ptp enic scsi_tgt pps_core i2c_algo_bit wmi
  [ 871.132967] CPU: 7 PID: 0 Comm: swapper/7 Not tainted 3.13.0-29-generic #53-Ubuntu
  [ 871.141500] Hardware name: Cisco Systems Inc UCSC-C220-M3S/UCSC-C220-M3S, BIOS C220M3.1.5.4f.0.111320130449 11/13/2013
  [ 871.153541] task: ffff881fd2d117f0 ti: ffff881fd2d18000 task.ti: ffff881fd2d18000
  [ 871.161975] RIP: 0010:[<ffffffff81612a6a>] [<ffffffff81612a6a>] skb_segment+0x95a/0x980
  [ 871.171124] RSP: 0018:ffff881fffce3498 EFLAGS: 00010206
  [ 871.177099] RAX: 0000000000000000 RBX: ffff881fb8929700 RCX: ffff881fcfba32f0
  [ 871.185120] RDX: 0000000000000050 RSI: ffff881fcfba3200 RDI: ffff881fcfba2200
  [ 871.193140] RBP: ffff881fffce3560 R08: 0000000000000042 R09: 0000000000000000
  [ 871.201160] R10: ffff881fb8929e00 R11: 00000000000005ea R12: ffff881fcfba22f0
  [ 871.209171] R13: 0000000000000000 R14: ffff881fd09a8400 R15: 0000000000000050
  [ 871.217191] FS: 0000000000000000(0000) GS:ffff881fffce0000(0000) knlGS:0000000000000000
  [ 871.226307] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  [ 871.232772] CR2: 00007fbd1ccb4000 CR3: 0000000001c0e000 CR4: 00000000001407e0
  [ 871.240793] Stack:
  [ 871.243079] 0000000000000000 0000000000000000 0000000000000042 ffffffffffffffbe
  [ 871.251470] 00000001000005ea 0000000000000040 ffff881fb8929e00 0000000100000000
  [ 871.259864] ffffffffffffffce 0000000000000074 00000032000005a8 ffff881fb8929700
  [ 871.268257] Call Trace:
  [ 871.271032] <IRQ>
  [ 871.273193]
  [ 871.274913] [<ffffffff8167cfbd>] tcp_gso_segment+0x10d/0x3f0
  [ 871.279926] [<ffffffffa03740f0>] ? vxlan_set_owner+0x60/0x60 [vxlan]
  [ 871.287178] [<ffffffff8168d422>] inet_gso_segment+0x132/0x360
  [ 871.293749] [<ffffffffa045c701>] ? bond_mode_name+0x21/0x30 [bonding]
  [ 871.301097] [<ffffffff8161fdac>] skb_mac_gso_segment+0x9c/0x180
  [ 871.307858] [<ffffffff816840d7>] skb_udp_tunnel_segment+0xd7/0x390
  [ 871.314906] [<ffffffff816847f0>] udp4_ufo_fragment+0x120/0x130
  [ 871.321566] [<ffffffff8168d422>] inet_gso_segment+0x132/0x360
  [ 871.328132] [<ffffffffa0563f67>] ? ipt_do_table+0x317/0x6aa [ip_tables]
  [ 871.335669] [<ffffffffa05a3500>] ? nf_conntrack_hash_check_insert+0x280/0x2e0 [nf_conntrack]
  [ 871.345268] [<ffffffff8161fdac>] skb_mac_gso_segment+0x9c/0x180
  [ 871.352026] [<ffffffff8161feed>] __skb_gso_segment+0x5d/0xb0
  [ 871.358494] [<ffffffff816201fa>] dev_hard_start_xmit+0x18a/0x560
  [ 871.365351] [<ffffffff816208e8>] __dev_queue_xmit+0x318/0x500
  [ 871.371910] [<ffffffffa05c0238>] ? ipv4_confirm+0x78/0x100 [nf_conntrack_ipv4]
  [ 871.380156] [<ffffffff8163e03b>] ? eth_header+0x2b/0xd0
  [ 871.386135] [<ffffffff81620ae0>] dev_queue_xmit+0x10/0x20
  [ 871.392315] [<ffffffff81629367>] neigh_connected_output+0xb7/0x100
  [ 871.399366] [<ffffffff81658eb0>] ip_finish_output+0x1b0/0x3b0
  [ 871.405922] [<ffffffff8165a418>] ip_output+0x58/0x90
  [ 871.411609] [<ffffffff81659b75>] ip_local_out+0x25/0x30
  [ 871.417596] [<ffffffff8169c43e>] iptunnel_xmit+0xee/0x110
  [ 871.423772] [<ffffffffa0376581>] vxlan_xmit_skb+0x1d1/0x350 [vxlan]
  [ 871.430928] [<ffffffffa055956b>] vxlan_tnl_send+0x11b/0x190 [openvswitch]
  [ 871.438666] [<ffffffffa055895d>] ovs_vport_send+0x1d/0x80 [openvswitch]
  [ 871.446203] [<ffffffffa054f19a>] do_output+0x2a/0x50 [openvswitch]
  [ 871.453255] [<ffffffffa054f643>] do_execute_actions+0x2e3/0xa90 [openvswitch]
  [ 871.461394] [<ffffffff8163f1ff>] ? sch_direct_xmit+0x5f/0x1c0
  [ 871.467960] [<ffffffffa054fe1b>] ovs_execute_actions+0x2b/0x30 [openvswitch]
  [ 871.475979] [<ffffffffa05526e2>] ovs_dp_process_received_packet+0x92/0x120 [openvswitch]
  [ 871.485198] [<ffffffff8136ced1>] ? csum_partial+0x11/0x20
  [ 871.491376] [<ffffffffa055889a>] ovs_vport_receive+0x2a/0x30 [openvswitch]
  [ 871.499206] [<ffffffffa0558ced>] internal_dev_xmit+0x1d/0x30 [openvswitch]
  [ 871.507035] [<ffffffff81620388>] dev_hard_start_xmit+0x318/0x560
  [ 871.513890] [<ffffffff816208e8>] __dev_queue_xmit+0x318/0x500
  [ 871.520454] [<ffffffff81620ae0>] dev_queue_xmit+0x10/0x20
  [ 871.526630] [<ffffffff81658fc9>] ip_finish_output+0x2c9/0x3b0
  [ 871.533193] [<ffffffff8165a418>] ip_output+0x58/0x90
  [ 871.538884] [<ffffffff8165639b>] ip_forward_finish+0x8b/0x170
  [ 871.545446] [<ffffffff816567d5>] ip_forward+0x355/0x410
  [ 871.551427] [<ffffffff816544ed>] ip_rcv_finish+0x7d/0x350
  [ 871.557600] [<ffffffff81654e38>] ip_rcv+0x298/0x3d0
  [ 871.563191] [<ffffffff8161e9b6>] __netif_receive_skb_core+0x666/0x840
  [ 871.570532] [<ffffffff8161eba8>] __netif_receive_skb+0x18/0x60
  [ 871.586559] [<ffffffff8161f6ce>] process_backlog+0xae/0x1a0
  [ 871.592928] [<ffffffff8161ef92>] net_rx_action+0x152/0x250
  [ 871.599211] [<ffffffff8106caec>] __do_softirq+0xec/0x2c0
  [ 871.605288] [<ffffffff8106d035>] irq_exit+0x105/0x110
  [ 871.611081] [<ffffffff8172cf16>] do_IRQ+0x56/0xc0
  [ 871.616485] [<ffffffff817226ad>] common_interrupt+0x6d/0x6d
  [ 871.622850] <EOI>
  [ 871.625006]
  [ 871.626727] [<ffffffff815cd0f2>] ? cpuidle_enter_state+0x52/0xc0
  [ 871.632122] [<ffffffff815cd219>] cpuidle_idle_call+0xb9/0x1f0
  [ 871.638694] [<ffffffff8101ce9e>] arch_cpu_idle+0xe/0x30
  [ 871.644681] [<ffffffff810beb95>] cpu_startup_entry+0xc5/0x290
  [ 871.651257] [<ffffffff81040fb8>] start_secondary+0x218/0x2c0
  [ 871.657722] Code: 0f 84 0c ff ff ff 8b 54 24 50 48 c7 c7 18 ae ae 81 44 89 ce 31 c0 e8 75 14 10 00 48 8b 7c 24 58 e9 ff fe ff ff 0f 1f 40 00 0f 0b <0f> 0b 0f 0b e8 d0 a5 10 00 48 c7 c0 ea f
  f ff ff e9 c7 fd ff ff
  [ 871.679616] RIP [<ffffffff81612a6a>] skb_segment+0x95a/0x980
  [ 871.686097] RSP <ffff881fffce3498>
  
  Fix has been committed in the upstream linux tree at
  5882a07c72093dc3a18e2d2b129fb200686bb6ee
  
  Netdev conversation can be viewed here.
  http://patchwork.ozlabs.org/patch/357291/

** Changed in: linux (Ubuntu)
     Assignee: (unassigned) => Dave Chiluk (chiluk)

** Changed in: linux (Ubuntu)
       Status: Triaged => In Progress

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1331219

Title:
  Kernel Bug at skb_segment + 0x95a

Status in “linux” package in Ubuntu:
  In Progress

Bug description:
  SRU Justification:
  Impact: A crash occurs with VXLAN packets when using GSO capable hardware such as the case with openstack neutron.

  Fix: Upstream fix 5882a07c7 fixes a locking error that previously allowed
  certain timings to hit this bug.

  Testcase: Create and run a Neutron gateway with VXLANs, and stress the
  network for an extended period of time.

  
  Kernel Bug affects 3.11-3.15 on machines using gso offload.

  Stack trace
  [ 871.025524] ------------[ cut here ]------------
  [ 871.030742] kernel BUG at /build/buildd/linux-3.13.0/net/core/skbuff.c:2903!
  [ 871.038669] invalid opcode: 0000 [#1] SMP
  [ 871.043318] Modules linked in: xt_nat xt_REDIRECT dccp_diag dccp tcp_diag udp_diag inet_diag unix_diag ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conn
  track nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp bridge ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables veth openvswitch gre vxlan ip_tunnel
   dm_crypt 8021q garp stp mrp llc bonding nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache sb_edac gpio_ich joydev edac_core x86_pkg_temp_thermal intel_powerclamp coretemp lpc_ich acpi_power_m
  eter kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd ipmi_si mac_hid lp parport btrfs xor raid6_pq libcrc32c
  hid_generic qla2xxx fnic libfcoe megaraid_sas
  [ 871.124025] igb usbhid libfc hid dca scsi_transport_fc ptp enic scsi_tgt pps_core i2c_algo_bit wmi
  [ 871.132967] CPU: 7 PID: 0 Comm: swapper/7 Not tainted 3.13.0-29-generic #53-Ubuntu
  [ 871.141500] Hardware name: Cisco Systems Inc UCSC-C220-M3S/UCSC-C220-M3S, BIOS C220M3.1.5.4f.0.111320130449 11/13/2013
  [ 871.153541] task: ffff881fd2d117f0 ti: ffff881fd2d18000 task.ti: ffff881fd2d18000
  [ 871.161975] RIP: 0010:[<ffffffff81612a6a>] [<ffffffff81612a6a>] skb_segment+0x95a/0x980
  [ 871.171124] RSP: 0018:ffff881fffce3498 EFLAGS: 00010206
  [ 871.177099] RAX: 0000000000000000 RBX: ffff881fb8929700 RCX: ffff881fcfba32f0
  [ 871.185120] RDX: 0000000000000050 RSI: ffff881fcfba3200 RDI: ffff881fcfba2200
  [ 871.193140] RBP: ffff881fffce3560 R08: 0000000000000042 R09: 0000000000000000
  [ 871.201160] R10: ffff881fb8929e00 R11: 00000000000005ea R12: ffff881fcfba22f0
  [ 871.209171] R13: 0000000000000000 R14: ffff881fd09a8400 R15: 0000000000000050
  [ 871.217191] FS: 0000000000000000(0000) GS:ffff881fffce0000(0000) knlGS:0000000000000000
  [ 871.226307] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  [ 871.232772] CR2: 00007fbd1ccb4000 CR3: 0000000001c0e000 CR4: 00000000001407e0
  [ 871.240793] Stack:
  [ 871.243079] 0000000000000000 0000000000000000 0000000000000042 ffffffffffffffbe
  [ 871.251470] 00000001000005ea 0000000000000040 ffff881fb8929e00 0000000100000000
  [ 871.259864] ffffffffffffffce 0000000000000074 00000032000005a8 ffff881fb8929700
  [ 871.268257] Call Trace:
  [ 871.271032] <IRQ>
  [ 871.273193]
  [ 871.274913] [<ffffffff8167cfbd>] tcp_gso_segment+0x10d/0x3f0
  [ 871.279926] [<ffffffffa03740f0>] ? vxlan_set_owner+0x60/0x60 [vxlan]
  [ 871.287178] [<ffffffff8168d422>] inet_gso_segment+0x132/0x360
  [ 871.293749] [<ffffffffa045c701>] ? bond_mode_name+0x21/0x30 [bonding]
  [ 871.301097] [<ffffffff8161fdac>] skb_mac_gso_segment+0x9c/0x180
  [ 871.307858] [<ffffffff816840d7>] skb_udp_tunnel_segment+0xd7/0x390
  [ 871.314906] [<ffffffff816847f0>] udp4_ufo_fragment+0x120/0x130
  [ 871.321566] [<ffffffff8168d422>] inet_gso_segment+0x132/0x360
  [ 871.328132] [<ffffffffa0563f67>] ? ipt_do_table+0x317/0x6aa [ip_tables]
  [ 871.335669] [<ffffffffa05a3500>] ? nf_conntrack_hash_check_insert+0x280/0x2e0 [nf_conntrack]
  [ 871.345268] [<ffffffff8161fdac>] skb_mac_gso_segment+0x9c/0x180
  [ 871.352026] [<ffffffff8161feed>] __skb_gso_segment+0x5d/0xb0
  [ 871.358494] [<ffffffff816201fa>] dev_hard_start_xmit+0x18a/0x560
  [ 871.365351] [<ffffffff816208e8>] __dev_queue_xmit+0x318/0x500
  [ 871.371910] [<ffffffffa05c0238>] ? ipv4_confirm+0x78/0x100 [nf_conntrack_ipv4]
  [ 871.380156] [<ffffffff8163e03b>] ? eth_header+0x2b/0xd0
  [ 871.386135] [<ffffffff81620ae0>] dev_queue_xmit+0x10/0x20
  [ 871.392315] [<ffffffff81629367>] neigh_connected_output+0xb7/0x100
  [ 871.399366] [<ffffffff81658eb0>] ip_finish_output+0x1b0/0x3b0
  [ 871.405922] [<ffffffff8165a418>] ip_output+0x58/0x90
  [ 871.411609] [<ffffffff81659b75>] ip_local_out+0x25/0x30
  [ 871.417596] [<ffffffff8169c43e>] iptunnel_xmit+0xee/0x110
  [ 871.423772] [<ffffffffa0376581>] vxlan_xmit_skb+0x1d1/0x350 [vxlan]
  [ 871.430928] [<ffffffffa055956b>] vxlan_tnl_send+0x11b/0x190 [openvswitch]
  [ 871.438666] [<ffffffffa055895d>] ovs_vport_send+0x1d/0x80 [openvswitch]
  [ 871.446203] [<ffffffffa054f19a>] do_output+0x2a/0x50 [openvswitch]
  [ 871.453255] [<ffffffffa054f643>] do_execute_actions+0x2e3/0xa90 [openvswitch]
  [ 871.461394] [<ffffffff8163f1ff>] ? sch_direct_xmit+0x5f/0x1c0
  [ 871.467960] [<ffffffffa054fe1b>] ovs_execute_actions+0x2b/0x30 [openvswitch]
  [ 871.475979] [<ffffffffa05526e2>] ovs_dp_process_received_packet+0x92/0x120 [openvswitch]
  [ 871.485198] [<ffffffff8136ced1>] ? csum_partial+0x11/0x20
  [ 871.491376] [<ffffffffa055889a>] ovs_vport_receive+0x2a/0x30 [openvswitch]
  [ 871.499206] [<ffffffffa0558ced>] internal_dev_xmit+0x1d/0x30 [openvswitch]
  [ 871.507035] [<ffffffff81620388>] dev_hard_start_xmit+0x318/0x560
  [ 871.513890] [<ffffffff816208e8>] __dev_queue_xmit+0x318/0x500
  [ 871.520454] [<ffffffff81620ae0>] dev_queue_xmit+0x10/0x20
  [ 871.526630] [<ffffffff81658fc9>] ip_finish_output+0x2c9/0x3b0
  [ 871.533193] [<ffffffff8165a418>] ip_output+0x58/0x90
  [ 871.538884] [<ffffffff8165639b>] ip_forward_finish+0x8b/0x170
  [ 871.545446] [<ffffffff816567d5>] ip_forward+0x355/0x410
  [ 871.551427] [<ffffffff816544ed>] ip_rcv_finish+0x7d/0x350
  [ 871.557600] [<ffffffff81654e38>] ip_rcv+0x298/0x3d0
  [ 871.563191] [<ffffffff8161e9b6>] __netif_receive_skb_core+0x666/0x840
  [ 871.570532] [<ffffffff8161eba8>] __netif_receive_skb+0x18/0x60
  [ 871.586559] [<ffffffff8161f6ce>] process_backlog+0xae/0x1a0
  [ 871.592928] [<ffffffff8161ef92>] net_rx_action+0x152/0x250
  [ 871.599211] [<ffffffff8106caec>] __do_softirq+0xec/0x2c0
  [ 871.605288] [<ffffffff8106d035>] irq_exit+0x105/0x110
  [ 871.611081] [<ffffffff8172cf16>] do_IRQ+0x56/0xc0
  [ 871.616485] [<ffffffff817226ad>] common_interrupt+0x6d/0x6d
  [ 871.622850] <EOI>
  [ 871.625006]
  [ 871.626727] [<ffffffff815cd0f2>] ? cpuidle_enter_state+0x52/0xc0
  [ 871.632122] [<ffffffff815cd219>] cpuidle_idle_call+0xb9/0x1f0
  [ 871.638694] [<ffffffff8101ce9e>] arch_cpu_idle+0xe/0x30
  [ 871.644681] [<ffffffff810beb95>] cpu_startup_entry+0xc5/0x290
  [ 871.651257] [<ffffffff81040fb8>] start_secondary+0x218/0x2c0
  [ 871.657722] Code: 0f 84 0c ff ff ff 8b 54 24 50 48 c7 c7 18 ae ae 81 44 89 ce 31 c0 e8 75 14 10 00 48 8b 7c 24 58 e9 ff fe ff ff 0f 1f 40 00 0f 0b <0f> 0b 0f 0b e8 d0 a5 10 00 48 c7 c0 ea f
  f ff ff e9 c7 fd ff ff
  [ 871.679616] RIP [<ffffffff81612a6a>] skb_segment+0x95a/0x980
  [ 871.686097] RSP <ffff881fffce3498>

  Fix has been committed in the upstream linux tree at
  5882a07c72093dc3a18e2d2b129fb200686bb6ee

  Netdev conversation can be viewed here.
  http://patchwork.ozlabs.org/patch/357291/

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1331219/+subscriptions
References

[Bug 1331219] [NEW] Kernel Bug at skb_segment + 0x95a
From: Dave Chiluk, 2014-06-17