← Back to team overview

kernel-packages team mailing list archive

[Bug 1579943] [NEW] Kernel OOPS

 

Public bug reported:

I am receiving kernel OOPS while testing openvswitch with openflow to
redirect traffic to a proxy.

Things work fine if I run a slow tcp connection rate (10conn/s) on our
test framework.

As soon as I increase the tcp connection rate (1000 conn/s) the kernel
OOPS almost immediately.

Here are a couple of the kernel OOPS:

===============================================================
ovs1604 login: [  264.174741] BUG: unable to handle kernel paging request at ffffffffb99e2b0c
[  264.518279] IP: [<ffffffff8171615d>] __netdev_pick_tx+0x8d/0x140
[  264.813723] PGD 2e0d067 PUD 2e0e063 PMD 0 
[  265.013608] Oops: 0000 [#1] SMP 
[  265.170788] Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 8021q garp mrp stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip_tables x_tables openvswitch nf_defrag_ipv6 nf_conntrack nls_iso8859_1 ipmi_ssif intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass input_leds 8250_fintek sb_edac edac_core ipmi_si hpilo ipmi_msghandler lpc_ich ioatdma shpchp acpi_power_meter mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic crct10dif_pclmul crc32_pclmul usbhid hid ixgbe aesni_intel dca aes_x86_64 vxlan lrw gf128mul ip6_udp_tunnel tg3 glue_helper udp_tunnel hpsa ablk_helper ptp cryptd pps_core scsi_transport_sas mdio wmi fjes
[  269.038220] CPU: 12 PID: 0 Comm: swapper/12 Not tainted 4.4.0-22-generic #39-Ubuntu
[  269.411556] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015
[  269.729466] task: ffff881038ba1b80 ti: ffff881038bb0000 task.ti: ffff881038bb0000
[  269.952910] ------------[ cut here ]------------
[  269.952918] WARNING: CPU: 0 PID: 4 at /build/linux-UbQGH5/linux-4.4.0/net/sched/sch_generic.c:303 dev_watchdog+0x237/0x240()
[  269.952919] NETDEV WATCHDOG: eth5 (ixgbe): transmit queue 18 timed out
[  269.952972] Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 8021q garp mrp stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip_tables x_tables openvswitch nf_defrag_ipv6 nf_conntrack nls_iso8859_1 ipmi_ssif intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass input_leds 8250_fintek sb_edac edac_core ipmi_si hpilo ipmi_msghandler lpc_ich ioatdma shpchp acpi_power_meter mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic crct10dif_pclmul crc32_pclmul usbhid hid ixgbe aesni_intel dca aes_x86_64 vxlan lrw gf128mul ip6_udp_tunnel tg3 glue_helper udp_tunnel hpsa ablk_helper ptp cryptd pps_core scsi_transport_sas mdio wmi fjes
[  269.952984] CPU: 0 PID: 4 Comm: kworker/0:0 Not tainted 4.4.0-22-generic #39-Ubuntu
[  269.952985] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015
[  269.952993] Workqueue: events dbs_timer
[  269.952997]  0000000000000286 00000000d6830222 ffff88103f603d98 ffffffff813e9c53
[  269.953000]  ffff88103f603de0 ffffffff81d62a90 ffff88103f603dd0 ffffffff81080fb2
[  269.953004]  0000000000000012 ffff8810309ddf00 0000000000000000 ffff881030720000
[  269.953005] Call Trace:
[  269.953015]  <IRQ>  [<ffffffff813e9c53>] dump_stack+0x63/0x90
[  269.953022]  [<ffffffff81080fb2>] warn_slowpath_common+0x82/0xc0
[  269.953025]  [<ffffffff8108104c>] warn_slowpath_fmt+0x5c/0x80
[  269.953030]  [<ffffffff81741df7>] dev_watchdog+0x237/0x240
[  269.953032]  [<ffffffff81741bc0>] ? qdisc_rcu_free+0x40/0x40
[  269.953039]  [<ffffffff810ec3b5>] call_timer_fn+0x35/0x120
[  269.953041]  [<ffffffff81741bc0>] ? qdisc_rcu_free+0x40/0x40
[  269.953045]  [<ffffffff810ecd6a>] run_timer_softirq+0x23a/0x2f0
[  269.953050]  [<ffffffff810859f1>] __do_softirq+0x101/0x290
[  269.953054]  [<ffffffff81085cf3>] irq_exit+0xa3/0xb0
[  269.953061]  [<ffffffff81827da2>] smp_apic_timer_interrupt+0x42/0x50
[  269.953064]  [<ffffffff81826062>] apic_timer_interrupt+0x82/0x90
[  269.953070]  <EOI>  [<ffffffff810ca84d>] ? native_queued_spin_lock_slowpath+0x10d/0x170
[  269.953075]  [<ffffffff81825061>] _raw_spin_lock+0x21/0x30
[  269.953079]  [<ffffffff816b9d23>] pcc_cpufreq_target+0x73/0x220
[  269.953083]  [<ffffffff816afe38>] __cpufreq_driver_target+0x88/0x370
[  269.953089]  [<ffffffff810fe28a>] ? get_cpu_idle_time_us+0x5a/0xd0
[  269.953093]  [<ffffffff816b3fe1>] od_check_cpu+0xc1/0xd0
[  269.953096]  [<ffffffff816b5d6e>] dbs_check_cpu+0x16e/0x1d0
[  269.953100]  [<ffffffff816b42a2>] od_dbs_timer+0xa2/0xf0
[  269.953103]  [<ffffffff816b5fa4>] dbs_timer+0xb4/0xe0
[  269.953109]  [<ffffffff8109a052>] process_one_work+0x162/0x480
[  269.953112]  [<ffffffff8109a3bb>] worker_thread+0x4b/0x4c0
[  269.953115]  [<ffffffff8109a370>] ? process_one_work+0x480/0x480
[  269.953119]  [<ffffffff810a0588>] kthread+0xd8/0xf0
[  269.953122]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[  269.953124]  [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[  269.953127]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[  269.953129] ---[ end trace 23deee586868b6e2 ]---
[  269.953134] ixgbe 0000:05:00.1 eth5: initiating reset due to tx timeout
[  270.688938] ixgbe 0000:05:00.0 eth4: initiating reset due to tx timeout
[  270.704938] ixgbe 0000:0b:00.0 eth6: initiating reset due to tx timeout
[  274.961065] ixgbe 0000:05:00.1 eth5: initiating reset due to tx timeout
[  275.697087] ixgbe 0000:05:00.0 eth4: initiating reset due to tx timeout
[  275.697089] ixgbe 0000:0b:00.0 eth6: initiating reset due to tx timeout
[  279.953212] ixgbe 0000:05:00.1 eth5: initiating reset due to tx timeout
[  280.689235] ixgbe 0000:05:00.0 eth4: initiating reset due to tx timeout
[  280.705234] ixgbe 0000:0b:00.0 eth6: initiating reset due to tx timeout
[  284.961358] ixgbe 0000:05:00.1 eth5: initiating reset due to tx timeout
[  285.697381] ixgbe 0000:05:00.0 eth4: initiating reset due to tx timeout
[  285.697383] ixgbe 0000:0b:00.0 eth6: initiating reset due to tx timeout
[  288.421453] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/1:1:365]
[  288.421500] Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 8021q garp mrp stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip_tables x_tables openvswitch nf_defrag_ipv6 nf_conntrack nls_iso8859_1 ipmi_ssif intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass input_leds 8250_fintek sb_edac edac_core ipmi_si hpilo ipmi_msghandler lpc_ich ioatdma shpchp acpi_power_meter mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic crct10dif_pclmul crc32_pclmul usbhid hid ixgbe aesni_intel dca aes_x86_64 vxlan lrw gf128mul ip6_udp_tunnel tg3 glue_helper udp_tunnel hpsa ablk_helper ptp cryptd pps_core scsi_transport_sas mdio wmi fjes
[  288.421512] CPU: 1 PID: 365 Comm: kworker/1:1 Tainted: G        W       4.4.0-22-generic #39-Ubuntu
[  288.421513] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015
[  288.421520] Workqueue: events dbs_timer
[  288.421523] task: ffff881037a22940 ti: ffff881033cc4000 task.ti: ffff881033cc4000
[  288.421529] RIP: 0010:[<ffffffff810ca84d>]  [<ffffffff810ca84d>] native_queued_spin_lock_slowpath+0x10d/0x170
[  288.421531] RSP: 0018:ffff881033cc7c48  EFLAGS: 00000246
[  288.421533] RAX: 0000000000000000 RBX: 0000000008fd0680 RCX: ffff88103fa97840
[  288.421534] RDX: ffff88103f657840 RSI: 0000000000080000 RDI: ffffffff821ed04c
[  288.421536] RBP: ffff881033cc7c48 R08: 0000000000000101 R09: 0000000000000002
[  288.421537] R10: ffff882033502a20 R11: ffff882033502a00 R12: ffffe8efffe44ef0
[  288.421538] R13: ffff882033502a00 R14: 0000000000000001 R15: 0000000000000100
[  288.421541] FS:  0000000000000000(0000) GS:ffff88103f640000(0000) knlGS:0000000000000000
[  288.421542] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  288.421544] CR2: 00007f4b2f1f9c20 CR3: 0000000002e0a000 CR4: 00000000001406e0
[  288.421545] Stack:
[  288.421548]  ffff881033cc7c58 ffffffff81825061 ffff881033cc7cb0 ffffffff816b9d23
[  288.421551]  0000000000000001 ffff881033cc7d80 0014726000000100 ffff8802001702a0
[  288.421554]  000000008a595031 ffff882033502a00 00000000001702a0 0000000000000002
[  288.421554] Call Trace:
[  288.421561]  [<ffffffff81825061>] _raw_spin_lock+0x21/0x30
[  288.421564]  [<ffffffff816b9d23>] pcc_cpufreq_target+0x73/0x220
[  288.421567]  [<ffffffff816afe38>] __cpufreq_driver_target+0x88/0x370
[  288.421572]  [<ffffffff810fe28a>] ? get_cpu_idle_time_us+0x5a/0xd0
[  288.421575]  [<ffffffff816b3fe1>] od_check_cpu+0xc1/0xd0
[  288.421579]  [<ffffffff816b5d6e>] dbs_check_cpu+0x16e/0x1d0
[  288.421582]  [<ffffffff816b42a2>] od_dbs_timer+0xa2/0xf0
[  288.421585]  [<ffffffff816b5fa4>] dbs_timer+0xb4/0xe0
[  288.421589]  [<ffffffff8109a052>] process_one_work+0x162/0x480
[  288.421592]  [<ffffffff8109a3bb>] worker_thread+0x4b/0x4c0
[  288.421595]  [<ffffffff8109a370>] ? process_one_work+0x480/0x480
[  288.421598]  [<ffffffff8109a370>] ? process_one_work+0x480/0x480
[  288.421601]  [<ffffffff810a0588>] kthread+0xd8/0xf0
[  288.421604]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[  288.421606]  [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[  288.421609]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[  288.421640] Code: 12 48 c1 e9 0c 83 e8 01 83 e1 30 48 98 48 81 c1 40 78 01 00 48 03 0c c5 80 42 f3 81 48 89 11 8b 42 08 85 c0 75 0d f3 90 8b 42 08 <85> c0 74 f7 eb 02 f3 90 8b 0f 66 85 c9 75 f7 39 f1 75 0f 89 c8 
[  288.429453] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:0:20]
[  288.429497] Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 8021q garp mrp stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip_tables x_tables openvswitch nf_defrag_ipv6 nf_conntrack nls_iso8859_1 ipmi_ssif intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass input_leds 8250_fintek sb_edac edac_core ipmi_si hpilo ipmi_msghandler lpc_ich ioatdma shpchp acpi_power_meter mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic crct10dif_pclmul crc32_pclmul usbhid hid ixgbe aesni_intel dca aes_x86_64 vxlan lrw gf128mul ip6_udp_tunnel tg3 glue_helper udp_tunnel hpsa ablk_helper ptp cryptd pps_core scsi_transport_sas mdio wmi fjes
[  288.429508] CPU: 2 PID: 20 Comm: kworker/2:0 Tainted: G        W    L  4.4.0-22-generic #39-Ubuntu
[  288.429509] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015
[  288.429514] Workqueue: events dbs_timer
[  288.429516] task: ffff881038466e00 ti: ffff881038044000 task.ti: ffff881038044000
[  288.429521] RIP: 0010:[<ffffffff810ca84d>]  [<ffffffff810ca84d>] native_queued_spin_lock_slowpath+0x10d/0x170
[  288.429523] RSP: 0018:ffff881038047c48  EFLAGS: 00000246
[  288.429524] RAX: 0000000000000000 RBX: 000000000a52d280 RCX: ffff88103f717840
[  288.429526] RDX: ffff88103f697840 RSI: 00000000000c0000 RDI: ffffffff821ed04c
[  288.429527] RBP: ffff881038047c48 R08: 0000000000000101 R09: 0000000000000004
[  288.429528] R10: ffff882033502c20 R11: ffff882033502c00 R12: ffffe8efffe84ef0
[  288.429530] R13: ffff882033502c00 R14: 0000000000000002 R15: 0000000000000100
[  288.429532] FS:  0000000000000000(0000) GS:ffff88103f680000(0000) knlGS:0000000000000000
[  288.429533] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  288.429535] CR2: 0000000000604180 CR3: 0000000002e0a000 CR4: 00000000001406e0
[  288.429536] Stack:
[  288.429539]  ffff881038047c58 ffffffff81825061 ffff881038047cb0 ffffffff816b9d23
[  288.429542]  ffff88103f696d00 0000000000000020 0018817000000100 ffff8802001a6da0
[  288.429544]  0000000070d943b3 ffff882033502c00 00000000001a6da0 0000000000000002
[  288.429545] Call Trace:
[  288.429551]  [<ffffffff81825061>] _raw_spin_lock+0x21/0x30
[  288.429553]  [<ffffffff816b9d23>] pcc_cpufreq_target+0x73/0x220
[  288.429556]  [<ffffffff816afe38>] __cpufreq_driver_target+0x88/0x370
[  288.429559]  [<ffffffff810fe28a>] ? get_cpu_idle_time_us+0x5a/0xd0
[  288.429563]  [<ffffffff816b3fe1>] od_check_cpu+0xc1/0xd0
[  288.429566]  [<ffffffff816b5d6e>] dbs_check_cpu+0x16e/0x1d0
[  288.429569]  [<ffffffff816b42a2>] od_dbs_timer+0xa2/0xf0
[  288.429572]  [<ffffffff816b5fa4>] dbs_timer+0xb4/0xe0
[  288.429575]  [<ffffffff8109a052>] process_one_work+0x162/0x480
[  288.429578]  [<ffffffff8109a3bb>] worker_thread+0x4b/0x4c0
[  288.429581]  [<ffffffff8109a370>] ? process_one_work+0x480/0x480
[  288.429583]  [<ffffffff810a0588>] kthread+0xd8/0xf0
[  288.429586]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[  288.429588]  [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[  288.429590]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[  288.429620] Code: 12 48 c1 e9 0c 83 e8 01 83 e1 30 48 98 48 81 c1 40 78 01 00 48 03 0c c5 80 42 f3 81 48 89 11 8b 42 08 85 c0 75 0d f3 90 8b 42 08 <85> c0 74 f7 eb 02 f3 90 8b 0f 66 85 c9 75 f7 39 f1 75 0f 89 c8 
================================================================


and another:

================================================================

ovs1604 login: [ 1138.011941] BUG: unable to handle kernel NULL pointer dereference at 0000000000000400
[ 1138.394597] IP: [<ffffffff81716136>] __netdev_pick_tx+0x66/0x140
[ 1138.688303] PGD 0 
[ 1138.785779] Oops: 0000 [#1] SMP 
[ 1138.943399] Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 8021q garp mrp stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip_tables x_tables openvswitch nf_defrag_ipv6 nf_conntrack nls_iso8859_1 ipmi_ssif intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass input_leds sb_edac edac_core hpilo lpc_ich ipmi_si ipmi_msghandler 8250_fintek ioatdma shpchp acpi_power_meter mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic ixgbe crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 dca lrw vxlan gf128mul ip6_udp_tunnel glue_helper tg3 ablk_helper udp_tunnel cryptd hpsa usbhid ptp pps_core hid scsi_transport_sas mdio wmi fjes
[ 1142.819687] CPU: 29 PID: 0 Comm: swapper/29 Not tainted 4.4.0-22-generic #39-Ubuntu
[ 1143.193063] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015
[ 1143.511272] task: ffff882038bc2940 ti: ffff881038404000 task.ti: ffff881038404000
[ 1143.876349] RIP: 0010:[<ffffffff81716136>]  [<ffffffff81716136>] __netdev_pick_tx+0x66/0x140
[ 1144.289553] RSP: 0018:ffff88103f9c36b0  EFLAGS: 00010206
[ 1144.548342] RAX: ffff88202e15ec00 RBX: ffff88100dddd400 RCX: ffffffff817160d0
[ 1144.898612] RDX: 000000000000007e RSI: ffff88100dddd400 RDI: ffff88102fda0000
[ 1145.247504] RBP: ffff88103f9c36e0 R08: 0000000000000074 R09: ffff88100dbe2818
[ 1145.595710] R10: ffff88100deba06c R11: 00000000000001b0 R12: ffff88102fda0000
[ 1145.944483] R13: 0000000000000000 R14: 00000000ffffffff R15: 0000000000000400
[ 1146.293274] FS:  0000000000000000(0000) GS:ffff88103f9c0000(0000) knlGS:0000000000000000
[ 1146.688266] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1146.968958] CR2: 0000000000000400 CR3: 0000000002e0a000 CR4: 00000000001406e0
[ 1147.322813] Stack:
[ 1147.421064]  ffff881033968e00 ffff88100dddd400 ffff88102fda0000[ 1147.692532] ------------[ cut here ]------------
[ 1147.692542] WARNING: CPU: 24 PID: 0 at /build/linux-UbQGH5/linux-4.4.0/net/sched/sch_generic.c:303 dev_watchdog+0x237/0x240()
[ 1147.692543] NETDEV WATCHDOG: eth5 (ixgbe): transmit queue 13 timed out
[ 1147.692544] Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 8021q garp mrp stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip_tables x_tables openvswitch nf_defrag_ipv6 nf_conntrack nls_iso8859_1 ipmi_ssif intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass input_leds sb_edac edac_core hpilo lpc_ich ipmi_si ipmi_msghandler 8250_fintek ioatdma shpchp acpi_power_meter mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic ixgbe crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 dca lrw vxlan gf128mul ip6_udp_tunnel glue_helper tg3 ablk_helper udp_tunnel cryptd hpsa usbhid ptp pps_core hid scsi_transport_sas mdio wmi fjes
[ 1147.696361] CPU: 24 PID: 0 Comm: swapper/24 Not tainted 4.4.0-22-generic #39-Ubuntu
[ 1147.696362] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015
[ 1147.696364]  0000000000000286 f29cb0207d8e6510 ffff88203f083d98 ffffffff813e9c53
[ 1147.696367]  ffff88203f083de0 ffffffff81d62a90 ffff88203f083dd0 ffffffff81080fb2
[ 1147.696566]  000000000000000d ffff881030505f00 0000000000000018 ffff88102fda0000
[ 1147.696758] Call Trace:
[ 1147.696760]  <IRQ>  [<ffffffff813e9c53>] dump_stack+0x63/0x90
[ 1147.697762]  [<ffffffff81080fb2>] warn_slowpath_common+0x82/0xc0
[ 1147.697974]  [<ffffffff8108104c>] warn_slowpath_fmt+0x5c/0x80
[ 1147.698185]  [<ffffffff81741df7>] dev_watchdog+0x237/0x240
[ 1147.698188]  [<ffffffff81741bc0>] ? qdisc_rcu_free+0x40/0x40
[ 1147.698193]  [<ffffffff810ec3b5>] call_timer_fn+0x35/0x120
[ 1147.698195]  [<ffffffff81741bc0>] ? qdisc_rcu_free+0x40/0x40
[ 1147.698199]  [<ffffffff810ecd6a>] run_timer_softirq+0x23a/0x2f0
[ 1147.698203]  [<ffffffff810859f1>] __do_softirq+0x101/0x290
[ 1147.698206]  [<ffffffff81085cf3>] irq_exit+0xa3/0xb0
[ 1147.698213]  [<ffffffff81827da2>] smp_apic_timer_interrupt+0x42/0x50
[ 1147.698216]  [<ffffffff81826062>] apic_timer_interrupt+0x82/0x90
[ 1147.698217]  <EOI>  [<ffffffff816bc411>] ? cpuidle_enter_state+0x111/0x2b0
[ 1147.698226]  [<ffffffff816bc5e7>] cpuidle_enter+0x17/0x20
[ 1147.698233]  [<ffffffff810c3db2>] call_cpuidle+0x32/0x60
[ 1147.698236]  [<ffffffff816bc5c3>] ? cpuidle_select+0x13/0x20
[ 1147.698239]  [<ffffffff810c4070>] cpu_startup_entry+0x290/0x350
[ 1147.698245]  [<ffffffff81051714>] start_secondary+0x154/0x190
[ 1147.698247] ---[ end trace 640662c79e82e205 ]---
[ 1147.698252] ixgbe 0000:05:00.1 eth5: initiating reset due to tx timeout
[ 1147.698277] ixgbe 0000:05:00.1 eth5: Reset adapter
[ 1148.258500] ixgbe 0000:05:00.1 eth5: detected SFP+: 5
[ 1148.468518] ixgbe 0000:05:00.1 eth5: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 1161.520473]  ffff88102fda0000
[ 1161.710452]  0000000000000000 ffffffff817160d0 ffff88103f9c3718 ffffffffc0457ce2
[ 1162.090787]  0000000000000000 ffff88102fda0000 ffff88100dddd400 0000000000000000
[ 1162.466909] Call Trace:
[ 1162.616986]  <IRQ> 
[ 1162.710449]  [<ffffffff817160d0>] ? __skb_tx_hash+0xa0/0xa0
[ 1163.027595]  [<ffffffffc0457ce2>] ixgbe_select_queue+0xb2/0x130 [ixgbe]
[ 1163.366984]  [<ffffffff8171d821>] netdev_pick_tx+0x61/0xe0
[ 1163.658701]  [<ffffffff8171d967>] __dev_queue_xmit+0xc7/0x590
[ 1163.955472]  [<ffffffff8171de40>] dev_queue_xmit+0x10/0x20
[ 1164.239831]  [<ffffffffc03567e8>] vlan_dev_hard_start_xmit+0x98/0x120 [8021q]
[ 1164.602400]  [<ffffffff8171d5d9>] dev_hard_start_xmit+0x249/0x3d0
[ 1164.914965]  [<ffffffff8171ddc6>] __dev_queue_xmit+0x526/0x590
[ 1165.218331]  [<ffffffff8171de40>] dev_queue_xmit+0x10/0x20
[ 1165.499745]  [<ffffffffc0332cba>] ovs_vport_send+0x4a/0xc0 [openvswitch]
[ 1165.839193]  [<ffffffffc03246a3>] do_output.isra.28+0x43/0x170 [openvswitch]
[ 1166.196050]  [<ffffffffc0325a09>] do_execute_actions+0x729/0x8b0 [openvswitch]
[ 1166.562138]  [<ffffffffc0325d73>] ovs_execute_actions+0x33/0xd0 [openvswitch]
[ 1166.921220]  [<ffffffffc0329564>] ovs_dp_process_packet+0x84/0x130 [openvswitch]
[ 1167.295247]  [<ffffffffc0325dee>] ovs_execute_actions+0xae/0xd0 [openvswitch]
[ 1167.654452]  [<ffffffffc0329564>] ovs_dp_process_packet+0x84/0x130 [openvswitch]
[ 1168.026460]  [<ffffffffc032a354>] ? key_extract+0x994/0xc80 [openvswitch]
[ 1168.371007]  [<ffffffffc03325ac>] ovs_vport_receive+0x6c/0xd0 [openvswitch]
[ 1168.722503]  [<ffffffff8171db26>] ? __dev_queue_xmit+0x286/0x590
[ 1169.027910]  [<ffffffffc01d16c0>] ? nf_nat_ipv4_fn+0x60/0x220 [nf_nat_ipv4]
[ 1169.378769]  [<ffffffff8175dd09>] ? ip_finish_output2+0x159/0x340
[ 1169.688345]  [<ffffffffc02fd23d>] ? ipv4_confirm+0x7d/0x100 [nf_conntrack_ipv4]
[ 1170.059626]  [<ffffffff8175ec56>] ? ip_finish_output+0x136/0x1f0
[ 1170.364133]  [<ffffffff817524d3>] ? nf_hook_slow+0x73/0xd0
[ 1170.639235]  [<ffffffff8175b2ed>] ? ip_forward_finish+0x4d/0x70
[ 1170.939768]  [<ffffffffc03337cd>] netdev_frame_hook+0xed/0x160 [openvswitch]
[ 1171.296146]  [<ffffffff8171af74>] __netif_receive_skb_core+0x364/0xa60
[ 1171.626731]  [<ffffffff81794900>] ? inet_gro_receive+0x180/0x200
[ 1171.931280]  [<ffffffff8171bf38>] ? dev_gro_receive+0x228/0x3c0
[ 1172.241496]  [<ffffffff8171b688>] __netif_receive_skb+0x18/0x60
[ 1172.541952]  [<ffffffff8171b702>] netif_receive_skb_internal+0x32/0xa0
[ 1172.621141] NMI watchdog: BUG: soft lockup - CPU#14 stuck for 23s! [kworker/14:2:2021]
[ 1172.621183] Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 8021q garp mrp stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip_tables x_tables openvswitch nf_defrag_ipv6 nf_conntrack nls_iso8859_1 ipmi_ssif intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass input_leds sb_edac edac_core hpilo lpc_ich ipmi_si ipmi_msghandler 8250_fintek ioatdma shpchp acpi_power_meter mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic ixgbe crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 dca lrw vxlan gf128mul ip6_udp_tunnel glue_helper tg3 ablk_helper udp_tunnel cryptd hpsa usbhid ptp pps_core hid scsi_transport_sas mdio wmi fjes
[ 1172.621194] CPU: 14 PID: 2021 Comm: kworker/14:2 Tainted: G        W       4.4.0-22-generic #39-Ubuntu
[ 1172.621195] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015
[ 1172.621202] Workqueue: events linkwatch_event
[ 1172.621204] task: ffff882032295280 ti: ffff882027830000 task.ti: ffff882027830000
[ 1172.621215] RIP: 0010:[<ffffffff81103a2d>]  [<ffffffff81103a2d>] smp_call_function_single+0x8d/0x130
[ 1172.621217] RSP: 0018:ffff882027833c40  EFLAGS: 00000202
[ 1172.621218] RAX: 0000000000000001 RBX: 000000000000001f RCX: 0000000000000000
[ 1172.621219] RDX: ffffffff81e52580 RSI: ffff88203ee17c00 RDI: 000000000000001f
[ 1172.621220] RBP: ffff882027833c88 R08: 0000000000000000 R09: 0000000000000000
[ 1172.621221] R10: 0000000000000004 R11: ffff8820322952e0 R12: ffffffff810e4d50
[ 1172.621222] R13: 0000000080000000 R14: 00dffeff5ff26fd7 R15: 000000000000001f
[ 1172.621224] FS:  0000000000000000(0000) GS:ffff88203ee00000(0000) knlGS:0000000000000000
[ 1172.621225] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1172.621226] CR2: 00007f7c0be726d0 CR3: 0000000002e0a000 CR4: 00000000001406e0
[ 1172.621227] Stack:
[ 1172.621230]  ffff88103f007480 ffff88100ddc6600 0000000000000000 0000000000000000
[ 1172.621232]  0000000000000000 0000000000000003 00000000b542f39f ffffffff81e52580
[ 1172.621234]  000000000000dca0 ffff882027833d08 ffffffff810e53d9 ffffffff81e52700
[ 1172.621234] Call Trace:
[ 1172.621240]  [<ffffffff810e53d9>] synchronize_sched_expedited+0x339/0x880
[ 1172.621244]  [<ffffffff810ec4f4>] ? lock_timer_base.isra.22+0x54/0x70
[ 1172.621247]  [<ffffffff810e59ae>] synchronize_rcu_expedited+0xe/0x10
[ 1172.621250]  [<ffffffff81716fd3>] synchronize_net+0x23/0x30
[ 1172.621253]  [<ffffffff817429eb>] dev_deactivate_many+0x23b/0x240
[ 1172.621255]  [<ffffffff81742a35>] dev_deactivate+0x45/0x70
[ 1172.621258]  [<ffffffff817310d0>] linkwatch_do_dev+0x30/0x50
[ 1172.621261]  [<ffffffff81731427>] __linkwatch_run_queue+0x167/0x190
[ 1172.621263]  [<ffffffff81731475>] linkwatch_event+0x25/0x30
[ 1172.621269]  [<ffffffff8109a052>] process_one_work+0x162/0x480
[ 1172.621271]  [<ffffffff8109a3bb>] worker_thread+0x4b/0x4c0
[ 1172.621274]  [<ffffffff8109a370>] ? process_one_work+0x480/0x480
[ 1172.621277]  [<ffffffff8109a370>] ? process_one_work+0x480/0x480
[ 1172.621280]  [<ffffffff810a0588>] kthread+0xd8/0xf0
[ 1172.621283]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 1172.621285]  [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[ 1172.621287]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 1172.621309] Code: c0 74 0c 9c 58 0f 1f 44 00 00 f6 c4 02 74 71 85 c9 75 48 48 c7 c6 00 7c 01 00 65 48 03 35 64 67 f0 7e 8b 46 18 a8 01 74 09 f3 90 <8b> 46 18 a8 01 75 f7 83 4e 18 01 48 89 d1 89 df 4c 89 e2 e8 3b 
[ 1189.966808]  [<ffffffff8171b85f>] napi_gro_complete+0x7f/0xa0
[ 1190.263283]  [<ffffffff8171b8da>] napi_gro_flush+0x5a/0x80
[ 1190.546867]  [<ffffffffc045a32e>] ixgbe_poll+0x58e/0x8d0 [ixgbe]
[ 1190.855476]  [<ffffffff8171bbce>] net_rx_action+0x21e/0x360
[ 1191.142638]  [<ffffffff810859f1>] __do_softirq+0x101/0x290
[ 1191.424157]  [<ffffffff81085cf3>] irq_exit+0xa3/0xb0
[ 1191.679965]  [<ffffffff81827ce4>] do_IRQ+0x54/0xd0
[ 1191.926377]  [<ffffffff81825dc2>] common_interrupt+0x82/0x82
[ 1192.228819]  <EOI> 
[ 1192.322267]  [<ffffffff816bc411>] ? cpuidle_enter_state+0x111/0x2b0
[ 1192.660739]  [<ffffffff816bc5e7>] cpuidle_enter+0x17/0x20
[ 1192.943489]  [<ffffffff810c3db2>] call_cpuidle+0x32/0x60
[ 1193.215091]  [<ffffffff816bc5c3>] ? cpuidle_select+0x13/0x20
[ 1193.499665]  [<ffffffff810c4070>] cpu_startup_entry+0x290/0x350
[ 1193.801670]  [<ffffffff81051714>] start_secondary+0x154/0x190
[ 1194.092982] Code: 87 e8 03 00 00 49 89 fc 48 85 c0 0f 84 da 00 00 00 8b 96 ac 00 00 00 83 ea 01 48 8d 44 d0 10 4c 8b 38 4d 85 ff 0f 84 c0 00 00 00 <41> 8b 1f 83 fb 01 0f 84 8d 00 00 00 f6 86 91 00 00 00 30 0f 84 
[ 1195.043812] RIP  [<ffffffff81716136>] __netdev_pick_tx+0x66/0x140
[ 1195.352932]  RSP <ffff88103f9c36b0>
[ 1195.534578] CR2: 0000000000000400
[ 1195.742792] ---[ end trace 640662c79e82e206 ]---
[ 1195.981707] Kernel panic - not syncing: Fatal exception in interrupt
[ 1196.303793] Kernel Offset: disabled
[ 1196.493128] ---[ end Kernel panic - not syncing: Fatal exception in interrupt
[ 1196.854462] ------------[ cut here ]------------
[ 1197.090915] WARNING: CPU: 29 PID: 0 at /build/linux-UbQGH5/linux-4.4.0/arch/x86/kernel/smp.c:125 native_smp_send_reschedule+0x60/0x70()
[ 1197.706261] Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 8021q garp mrp stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip_tables x_tables openvswitch nf_defrag_ipv6 nf_conntrack nls_iso8859_1 ipmi_ssif intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass input_leds sb_edac edac_core hpilo lpc_ich ipmi_si ipmi_msghandler 8250_fintek ioatdma shpchp acpi_power_meter mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic ixgbe crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 dca lrw vxlan gf128mul ip6_udp_tunnel glue_helper tg3 ablk_helper udp_tunnel cryptd hpsa usbhid ptp pps_core hid scsi_transport_sas mdio wmi fjes
[ 1201.718155] CPU: 29 PID: 0 Comm: swapper/29 Tainted: G      D W    L  4.4.0-22-generic #39-Ubuntu
[ 1202.164963] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015
[ 1202.499892]  0000000000000086 f9c2da801aceb89c ffff88103f9c3168 ffffffff813e9c53
[ 1202.880066]  0000000000000000 ffffffff81ca35b0 ffff88103f9c31a0 ffffffff81080fb2
[ 1203.259124]  000000000000000f ffff88203ee56d00 000000000000001d ffff882038bc2940
[ 1203.636609] Call Trace:
[ 1203.773175]  <IRQ>  [<ffffffff813e9c53>] dump_stack+0x63/0x90
[ 1204.070769]  [<ffffffff81080fb2>] warn_slowpath_common+0x82/0xc0
[ 1204.383768]  [<ffffffff810810fa>] warn_slowpath_null+0x1a/0x20
[ 1204.685193]  [<ffffffff81050970>] native_smp_send_reschedule+0x60/0x70
[ 1205.020826]  [<ffffffff810be7a3>] trigger_load_balance+0x133/0x210
[ 1205.334824]  [<ffffffff810ad3a6>] scheduler_tick+0xa6/0xd0
[ 1205.619034]  [<ffffffff810fe560>] ? tick_sched_handle.isra.14+0x60/0x60
[ 1205.959021]  [<ffffffff810ee961>] update_process_times+0x51/0x60
[ 1206.270252]  [<ffffffff810fe525>] tick_sched_handle.isra.14+0x25/0x60
[ 1206.601615]  [<ffffffff810fe59d>] tick_sched_timer+0x3d/0x70
[ 1206.894935]  [<ffffffff810ef282>] __hrtimer_run_queues+0x102/0x290
[ 1207.209571]  [<ffffffff810efa48>] hrtimer_interrupt+0xa8/0x1a0
[ 1207.510638]  [<ffffffff81052fa8>] local_apic_timer_interrupt+0x38/0x60
[ 1207.845707]  [<ffffffff81827d9d>] smp_apic_timer_interrupt+0x3d/0x50
[ 1208.172391]  [<ffffffff81826062>] apic_timer_interrupt+0x82/0x90
[ 1208.482226]  [<ffffffff8118bbb9>] ? panic+0x1d5/0x215
[ 1208.745324]  [<ffffffff8118bbb2>] ? panic+0x1ce/0x215
[ 1209.008398]  [<ffffffff81031c4a>] oops_end+0xca/0xd0
[ 1209.263560]  [<ffffffff8106ad45>] no_context+0x135/0x380
[ 1209.538824]  [<ffffffff8106b010>] __bad_area_nosemaphore+0x80/0x1f0
[ 1209.860782]  [<ffffffff8106b193>] bad_area_nosemaphore+0x13/0x20
[ 1210.170022]  [<ffffffff8106b457>] __do_page_fault+0xb7/0x400
[ 1210.463123]  [<ffffffffc0462a02>] ? ixgbe_xmit_frame+0x42/0x90 [ixgbe]
[ 1210.797385]  [<ffffffff8106b7c2>] do_page_fault+0x22/0x30
[ 1211.076463]  [<ffffffff81827478>] page_fault+0x28/0x30
[ 1211.342835]  [<ffffffff817160d0>] ? __skb_tx_hash+0xa0/0xa0
[ 1211.633287]  [<ffffffff81716136>] ? __netdev_pick_tx+0x66/0x140
[ 1211.940942]  [<ffffffff817160d0>] ? __skb_tx_hash+0xa0/0xa0
[ 1212.230203]  [<ffffffffc0457ce2>] ixgbe_select_queue+0xb2/0x130 [ixgbe]
[ 1212.566347]  [<ffffffff8171d821>] netdev_pick_tx+0x61/0xe0
[ 1212.851473]  [<ffffffff8171d967>] __dev_queue_xmit+0xc7/0x590
[ 1213.152225]  [<ffffffff8171de40>] dev_queue_xmit+0x10/0x20
[ 1213.437482]  [<ffffffffc03567e8>] vlan_dev_hard_start_xmit+0x98/0x120 [8021q]
[ 1213.801585]  [<ffffffff8171d5d9>] dev_hard_start_xmit+0x249/0x3d0
[ 1214.115849]  [<ffffffff8171ddc6>] __dev_queue_xmit+0x526/0x590
[ 1214.416730]  [<ffffffff8171de40>] dev_queue_xmit+0x10/0x20
[ 1214.699175]  [<ffffffffc0332cba>] ovs_vport_send+0x4a/0xc0 [openvswitch]
[ 1215.038336]  [<ffffffffc03246a3>] do_output.isra.28+0x43/0x170 [openvswitch]
[ 1215.394550]  [<ffffffffc0325a09>] do_execute_actions+0x729/0x8b0 [openvswitch]
[ 1215.758636]  [<ffffffffc0325d73>] ovs_execute_actions+0x33/0xd0 [openvswitch]
[ 1216.118188]  [<ffffffffc0329564>] ovs_dp_process_packet+0x84/0x130 [openvswitch]
[ 1216.490791]  [<ffffffffc0325dee>] ovs_execute_actions+0xae/0xd0 [openvswitch]
[ 1216.849706]  [<ffffffffc0329564>] ovs_dp_process_packet+0x84/0x130 [openvswitch]
[ 1217.222344]  [<ffffffffc032a354>] ? key_extract+0x994/0xc80 [openvswitch]
[ 1217.566366]  [<ffffffffc03325ac>] ovs_vport_receive+0x6c/0xd0 [openvswitch]
[ 1217.918259]  [<ffffffff8171db26>] ? __dev_queue_xmit+0x286/0x590
[ 1218.222395]  [<ffffffffc01d16c0>] ? nf_nat_ipv4_fn+0x60/0x220 [nf_nat_ipv4]
[ 1218.572934]  [<ffffffff8175dd09>] ? ip_finish_output2+0x159/0x340
[ 1218.881368]  [<ffffffffc02fd23d>] ? ipv4_confirm+0x7d/0x100 [nf_conntrack_ipv4]
[ 1219.248697]  [<ffffffff8175ec56>] ? ip_finish_output+0x136/0x1f0
[ 1219.552761]  [<ffffffff817524d3>] ? nf_hook_slow+0x73/0xd0
[ 1219.832000]  [<ffffffff8175b2ed>] ? ip_forward_finish+0x4d/0x70
[ 1220.131812]  [<ffffffffc03337cd>] netdev_frame_hook+0xed/0x160 [openvswitch]
[ 1220.486445]  [<ffffffff8171af74>] __netif_receive_skb_core+0x364/0xa60
[ 1220.816076]  [<ffffffff81794900>] ? inet_gro_receive+0x180/0x200
[ 1221.119917]  [<ffffffff8171bf38>] ? dev_gro_receive+0x228/0x3c0
[ 1221.419505]  [<ffffffff8171b688>] __netif_receive_skb+0x18/0x60
[ 1221.718992]  [<ffffffff8171b702>] netif_receive_skb_internal+0x32/0xa0
[ 1222.048204]  [<ffffffff8171b85f>] napi_gro_complete+0x7f/0xa0
[ 1222.330734]  [<ffffffff8171b8da>] napi_gro_flush+0x5a/0x80
[ 1222.609174]  [<ffffffffc045a32e>] ixgbe_poll+0x58e/0x8d0 [ixgbe]
[ 1222.914199]  [<ffffffff8171bbce>] net_rx_action+0x21e/0x360
[ 1223.196818]  [<ffffffff810859f1>] __do_softirq+0x101/0x290
[ 1223.475092]  [<ffffffff81085cf3>] irq_exit+0xa3/0xb0
[ 1223.728540]  [<ffffffff81827ce4>] do_IRQ+0x54/0xd0
[ 1223.973680]  [<ffffffff81825dc2>] common_interrupt+0x82/0x82
[ 1224.260746]  <EOI>  [<ffffffff816bc411>] ? cpuidle_enter_state+0x111/0x2b0
[ 1224.607877]  [<ffffffff816bc5e7>] cpuidle_enter+0x17/0x20
[ 1224.886873]  [<ffffffff810c3db2>] call_cpuidle+0x32/0x60
[ 1225.158221]  [<ffffffff816bc5c3>] ? cpuidle_select+0x13/0x20
[ 1225.447849]  [<ffffffff810c4070>] cpu_startup_entry+0x290/0x350
[ 1225.748616]  [<ffffffff81051714>] start_secondary+0x154/0x190
[ 1226.047825] ---[ end trace 640662c79e82e207 ]---

=======================================================


I'm not sure if providing the current openvswitch configuration will help.  The setup is fairly complicated.  Here is the basic configuration:

# ovs-vsctl show
1a7ea432-8b96-445e-8734-d71d10533302
    Bridge "ovsbr0"
        Port "eth6"
            tag: 998
            Interface "eth6"
        Port "eth5.110"
            tag: 110
            Interface "eth5.110"
        Port "eth4"
            tag: 999
            Interface "eth4"
        Port int-servers
            tag: 998
            Interface int-servers
                type: internal
        Port client
            tag: 999
            Interface client
                type: internal
        Port "ovsbr0"
            Interface "ovsbr0"
                type: internal
        Port ingress
            tag: 110
            Interface ingress
                type: internal
        Port "eth1.110"
            tag: 110
            Interface "eth1.110"
    ovs_version: "2.5.0"


# ovs-ofctl -O Openflow15 dump-groups ovsbr0
OFPST_GROUP_DESC reply (OF1.5) (xid=0x2):
 group_id=200,type=select,selection_method=hash,fields(ip_dst,tcp_dst),bucket=bucket_id:0,actions=set_field:fa:16:3e:17:2f:f1->eth_dst,output:9,bucket=bucket_id:1,actions=set_field:fa:16:3e:2f:51:99->eth_dst,output:9,bucket=bucket_id:2,actions=set_field:fa:16:3e:76:f3:42->eth_dst,output:9,bucket=bucket_id:3,actions=set_field:fa:16:3e:c0:cf:10->eth_dst,output:9,bucket=bucket_id:4,actions=set_field:fa:16:3e:ba:6f:b7->eth_dst,output:9
 group_id=100,type=select,selection_method=hash,fields(ip_src,tcp_src),bucket=bucket_id:0,actions=set_field:fa:16:3e:17:2f:f1->eth_dst,output:9,bucket=bucket_id:1,actions=set_field:fa:16:3e:2f:51:99->eth_dst,output:9,bucket=bucket_id:2,actions=set_field:fa:16:3e:76:f3:42->eth_dst,output:9,bucket=bucket_id:3,actions=set_field:fa:16:3e:c0:cf:10->eth_dst,output:9,bucket=bucket_id:4,actions=set_field:fa:16:3e:ba:6f:b7->eth_dst,output:9

# ovs-ofctl -O Openflow15 dump-flows ovsbr0
OFPST_FLOW reply (OF1.5) (xid=0x2):
 cookie=0x0, duration=206.013s, table=0, n_packets=0, n_bytes=0, priority=100,in_port=2 actions=resubmit(,100)
 cookie=0x0, duration=206.013s, table=0, n_packets=0, n_bytes=0, priority=100,in_port=10 actions=resubmit(,101)
 cookie=0x0, duration=206.009s, table=0, n_packets=8, n_bytes=632, priority=100,in_port=9 actions=resubmit(,200)
 cookie=0x0, duration=206.013s, table=0, n_packets=8, n_bytes=560, priority=0 actions=NORMAL
 cookie=0x0, duration=206.012s, table=100, n_packets=0, n_bytes=0, priority=100,ct_state=-trk,tcp,nw_src=10.100.0.0/16,tp_dst=80 actions=ct(table=110)
 cookie=0x0, duration=206.012s, table=100, n_packets=0, n_bytes=0, priority=0 actions=NORMAL
 cookie=0x0, duration=206.012s, table=101, n_packets=0, n_bytes=0, priority=100,ct_state=-trk,tcp,nw_dst=10.100.0.0/16,tp_src=80 actions=ct(table=120,zone=200)
 cookie=0x0, duration=206.012s, table=101, n_packets=0, n_bytes=0, priority=0 actions=NORMAL
 cookie=0x0, duration=206.011s, table=110, n_packets=0, n_bytes=0, priority=100,ct_state=+new+trk,ip actions=ct(commit,exec(set_field:0x64->ct_mark)),group:100
 cookie=0x0, duration=206.010s, table=110, n_packets=0, n_bytes=0, priority=100,ct_state=+est-rpl+trk,ip actions=ct(commit),group:100
 cookie=0x0, duration=206.010s, table=110, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=206.010s, table=120, n_packets=0, n_bytes=0, priority=101,ct_state=+est+rpl+trk,ct_zone=200,ip actions=ct(commit,zone=200),group:200
 cookie=0x0, duration=206.010s, table=120, n_packets=0, n_bytes=0, priority=100,ct_state=+est+rpl+trk,ip actions=ct(commit,zone=200),group:200
 cookie=0x0, duration=206.009s, table=120, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=206.009s, table=200, n_packets=0, n_bytes=0, priority=100,ct_state=-trk,tcp,nw_src=10.100.0.0/16,tp_dst=80 actions=ct(table=210,zone=200)
 cookie=0x0, duration=206.008s, table=200, n_packets=0, n_bytes=0, priority=100,ct_state=-trk,tcp,nw_dst=10.100.0.0/16,tp_src=80 actions=ct(table=220)
 cookie=0x0, duration=206.008s, table=200, n_packets=8, n_bytes=632, priority=0 actions=NORMAL
 cookie=0x0, duration=206.008s, table=210, n_packets=0, n_bytes=0, priority=100,ct_state=+new+trk,ip actions=ct(commit,zone=200),NORMAL
 cookie=0x0, duration=206.007s, table=210, n_packets=0, n_bytes=0, priority=100,ct_state=+est-rpl+trk,ip actions=ct(commit,zone=200),NORMAL
 cookie=0x0, duration=206.007s, table=210, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=206.006s, table=220, n_packets=0, n_bytes=0, priority=100,ct_state=+est+rpl+trk,ip actions=ct(commit),NORMAL
 cookie=0x0, duration=206.006s, table=220, n_packets=0, n_bytes=0, priority=100,ct_state=+rpl+trk,ip actions=ct(commit),NORMAL
 cookie=0x0, duration=206.006s, table=220, n_packets=0, n_bytes=0, priority=0 actions=drop

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-4.4.0-22-generic 4.4.0-22.39
ProcVersionSignature: Ubuntu 4.4.0-22.39-generic 4.4.8
Uname: Linux 4.4.0-22-generic x86_64
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116,  1 May  9 17:05 seq
 crw-rw---- 1 root audio 116, 33 May  9 17:05 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.1-0ubuntu2
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
Date: Mon May  9 17:12:43 2016
HibernationDevice: RESUME=/dev/mapper/ovs1604--vg-swap
InstallationDate: Installed on 2016-05-09 (0 days ago)
InstallationMedia: Ubuntu-Server 16.04 LTS "Xenial Xerus" - Release amd64 (20160420.3)
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
MachineType: HP ProLiant DL380 Gen9
PciMultimedia:
 
ProcFB: 0 EFI VGA
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-22-generic.efi.signed root=/dev/mapper/hostname--vg-root ro net.ifnames=0 biosdevname=0 console=ttyS0,9600 console=tty1
RelatedPackageVersions:
 linux-restricted-modules-4.4.0-22-generic N/A
 linux-backports-modules-4.4.0-22-generic  N/A
 linux-firmware                            1.157
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 05/06/2015
dmi.bios.vendor: HP
dmi.bios.version: P89
dmi.chassis.type: 23
dmi.chassis.vendor: HP
dmi.modalias: dmi:bvnHP:bvrP89:bd05/06/2015:svnHP:pnProLiantDL380Gen9:pvr:cvnHP:ct23:cvr:
dmi.product.name: ProLiant DL380 Gen9
dmi.sys.vendor: HP

** Affects: linux (Ubuntu)
     Importance: Undecided
         Status: Confirmed


** Tags: amd64 apport-bug xenial

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1579943

Title:
  Kernel OOPS

Status in linux package in Ubuntu:
  Confirmed

Bug description:
  I am receiving kernel OOPS while testing openvswitch with openflow to
  redirect traffic to a proxy.

  Things work fine if I run a slow tcp connection rate (10conn/s) on our
  test framework.

  As soon as I increase the tcp connection rate (1000 conn/s) the kernel
  OOPS almost immediately.

  Here are a couple of the kernel OOPS:

  ===============================================================
  ovs1604 login: [  264.174741] BUG: unable to handle kernel paging request at ffffffffb99e2b0c
  [  264.518279] IP: [<ffffffff8171615d>] __netdev_pick_tx+0x8d/0x140
  [  264.813723] PGD 2e0d067 PUD 2e0e063 PMD 0 
  [  265.013608] Oops: 0000 [#1] SMP 
  [  265.170788] Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 8021q garp mrp stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip_tables x_tables openvswitch nf_defrag_ipv6 nf_conntrack nls_iso8859_1 ipmi_ssif intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass input_leds 8250_fintek sb_edac edac_core ipmi_si hpilo ipmi_msghandler lpc_ich ioatdma shpchp acpi_power_meter mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic crct10dif_pclmul crc32_pclmul usbhid hid ixgbe aesni_intel dca aes_x86_64 vxlan lrw gf128mul ip6_udp_tunnel tg3 glue_helper udp_tunnel hpsa ablk_helper ptp cryptd pps_core scsi_transport_sas mdio wmi fjes
  [  269.038220] CPU: 12 PID: 0 Comm: swapper/12 Not tainted 4.4.0-22-generic #39-Ubuntu
  [  269.411556] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015
  [  269.729466] task: ffff881038ba1b80 ti: ffff881038bb0000 task.ti: ffff881038bb0000
  [  269.952910] ------------[ cut here ]------------
  [  269.952918] WARNING: CPU: 0 PID: 4 at /build/linux-UbQGH5/linux-4.4.0/net/sched/sch_generic.c:303 dev_watchdog+0x237/0x240()
  [  269.952919] NETDEV WATCHDOG: eth5 (ixgbe): transmit queue 18 timed out
  [  269.952972] Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 8021q garp mrp stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip_tables x_tables openvswitch nf_defrag_ipv6 nf_conntrack nls_iso8859_1 ipmi_ssif intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass input_leds 8250_fintek sb_edac edac_core ipmi_si hpilo ipmi_msghandler lpc_ich ioatdma shpchp acpi_power_meter mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic crct10dif_pclmul crc32_pclmul usbhid hid ixgbe aesni_intel dca aes_x86_64 vxlan lrw gf128mul ip6_udp_tunnel tg3 glue_helper udp_tunnel hpsa ablk_helper ptp cryptd pps_core scsi_transport_sas mdio wmi fjes
  [  269.952984] CPU: 0 PID: 4 Comm: kworker/0:0 Not tainted 4.4.0-22-generic #39-Ubuntu
  [  269.952985] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015
  [  269.952993] Workqueue: events dbs_timer
  [  269.952997]  0000000000000286 00000000d6830222 ffff88103f603d98 ffffffff813e9c53
  [  269.953000]  ffff88103f603de0 ffffffff81d62a90 ffff88103f603dd0 ffffffff81080fb2
  [  269.953004]  0000000000000012 ffff8810309ddf00 0000000000000000 ffff881030720000
  [  269.953005] Call Trace:
  [  269.953015]  <IRQ>  [<ffffffff813e9c53>] dump_stack+0x63/0x90
  [  269.953022]  [<ffffffff81080fb2>] warn_slowpath_common+0x82/0xc0
  [  269.953025]  [<ffffffff8108104c>] warn_slowpath_fmt+0x5c/0x80
  [  269.953030]  [<ffffffff81741df7>] dev_watchdog+0x237/0x240
  [  269.953032]  [<ffffffff81741bc0>] ? qdisc_rcu_free+0x40/0x40
  [  269.953039]  [<ffffffff810ec3b5>] call_timer_fn+0x35/0x120
  [  269.953041]  [<ffffffff81741bc0>] ? qdisc_rcu_free+0x40/0x40
  [  269.953045]  [<ffffffff810ecd6a>] run_timer_softirq+0x23a/0x2f0
  [  269.953050]  [<ffffffff810859f1>] __do_softirq+0x101/0x290
  [  269.953054]  [<ffffffff81085cf3>] irq_exit+0xa3/0xb0
  [  269.953061]  [<ffffffff81827da2>] smp_apic_timer_interrupt+0x42/0x50
  [  269.953064]  [<ffffffff81826062>] apic_timer_interrupt+0x82/0x90
  [  269.953070]  <EOI>  [<ffffffff810ca84d>] ? native_queued_spin_lock_slowpath+0x10d/0x170
  [  269.953075]  [<ffffffff81825061>] _raw_spin_lock+0x21/0x30
  [  269.953079]  [<ffffffff816b9d23>] pcc_cpufreq_target+0x73/0x220
  [  269.953083]  [<ffffffff816afe38>] __cpufreq_driver_target+0x88/0x370
  [  269.953089]  [<ffffffff810fe28a>] ? get_cpu_idle_time_us+0x5a/0xd0
  [  269.953093]  [<ffffffff816b3fe1>] od_check_cpu+0xc1/0xd0
  [  269.953096]  [<ffffffff816b5d6e>] dbs_check_cpu+0x16e/0x1d0
  [  269.953100]  [<ffffffff816b42a2>] od_dbs_timer+0xa2/0xf0
  [  269.953103]  [<ffffffff816b5fa4>] dbs_timer+0xb4/0xe0
  [  269.953109]  [<ffffffff8109a052>] process_one_work+0x162/0x480
  [  269.953112]  [<ffffffff8109a3bb>] worker_thread+0x4b/0x4c0
  [  269.953115]  [<ffffffff8109a370>] ? process_one_work+0x480/0x480
  [  269.953119]  [<ffffffff810a0588>] kthread+0xd8/0xf0
  [  269.953122]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
  [  269.953124]  [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
  [  269.953127]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
  [  269.953129] ---[ end trace 23deee586868b6e2 ]---
  [  269.953134] ixgbe 0000:05:00.1 eth5: initiating reset due to tx timeout
  [  270.688938] ixgbe 0000:05:00.0 eth4: initiating reset due to tx timeout
  [  270.704938] ixgbe 0000:0b:00.0 eth6: initiating reset due to tx timeout
  [  274.961065] ixgbe 0000:05:00.1 eth5: initiating reset due to tx timeout
  [  275.697087] ixgbe 0000:05:00.0 eth4: initiating reset due to tx timeout
  [  275.697089] ixgbe 0000:0b:00.0 eth6: initiating reset due to tx timeout
  [  279.953212] ixgbe 0000:05:00.1 eth5: initiating reset due to tx timeout
  [  280.689235] ixgbe 0000:05:00.0 eth4: initiating reset due to tx timeout
  [  280.705234] ixgbe 0000:0b:00.0 eth6: initiating reset due to tx timeout
  [  284.961358] ixgbe 0000:05:00.1 eth5: initiating reset due to tx timeout
  [  285.697381] ixgbe 0000:05:00.0 eth4: initiating reset due to tx timeout
  [  285.697383] ixgbe 0000:0b:00.0 eth6: initiating reset due to tx timeout
  [  288.421453] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/1:1:365]
  [  288.421500] Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 8021q garp mrp stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip_tables x_tables openvswitch nf_defrag_ipv6 nf_conntrack nls_iso8859_1 ipmi_ssif intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass input_leds 8250_fintek sb_edac edac_core ipmi_si hpilo ipmi_msghandler lpc_ich ioatdma shpchp acpi_power_meter mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic crct10dif_pclmul crc32_pclmul usbhid hid ixgbe aesni_intel dca aes_x86_64 vxlan lrw gf128mul ip6_udp_tunnel tg3 glue_helper udp_tunnel hpsa ablk_helper ptp cryptd pps_core scsi_transport_sas mdio wmi fjes
  [  288.421512] CPU: 1 PID: 365 Comm: kworker/1:1 Tainted: G        W       4.4.0-22-generic #39-Ubuntu
  [  288.421513] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015
  [  288.421520] Workqueue: events dbs_timer
  [  288.421523] task: ffff881037a22940 ti: ffff881033cc4000 task.ti: ffff881033cc4000
  [  288.421529] RIP: 0010:[<ffffffff810ca84d>]  [<ffffffff810ca84d>] native_queued_spin_lock_slowpath+0x10d/0x170
  [  288.421531] RSP: 0018:ffff881033cc7c48  EFLAGS: 00000246
  [  288.421533] RAX: 0000000000000000 RBX: 0000000008fd0680 RCX: ffff88103fa97840
  [  288.421534] RDX: ffff88103f657840 RSI: 0000000000080000 RDI: ffffffff821ed04c
  [  288.421536] RBP: ffff881033cc7c48 R08: 0000000000000101 R09: 0000000000000002
  [  288.421537] R10: ffff882033502a20 R11: ffff882033502a00 R12: ffffe8efffe44ef0
  [  288.421538] R13: ffff882033502a00 R14: 0000000000000001 R15: 0000000000000100
  [  288.421541] FS:  0000000000000000(0000) GS:ffff88103f640000(0000) knlGS:0000000000000000
  [  288.421542] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  [  288.421544] CR2: 00007f4b2f1f9c20 CR3: 0000000002e0a000 CR4: 00000000001406e0
  [  288.421545] Stack:
  [  288.421548]  ffff881033cc7c58 ffffffff81825061 ffff881033cc7cb0 ffffffff816b9d23
  [  288.421551]  0000000000000001 ffff881033cc7d80 0014726000000100 ffff8802001702a0
  [  288.421554]  000000008a595031 ffff882033502a00 00000000001702a0 0000000000000002
  [  288.421554] Call Trace:
  [  288.421561]  [<ffffffff81825061>] _raw_spin_lock+0x21/0x30
  [  288.421564]  [<ffffffff816b9d23>] pcc_cpufreq_target+0x73/0x220
  [  288.421567]  [<ffffffff816afe38>] __cpufreq_driver_target+0x88/0x370
  [  288.421572]  [<ffffffff810fe28a>] ? get_cpu_idle_time_us+0x5a/0xd0
  [  288.421575]  [<ffffffff816b3fe1>] od_check_cpu+0xc1/0xd0
  [  288.421579]  [<ffffffff816b5d6e>] dbs_check_cpu+0x16e/0x1d0
  [  288.421582]  [<ffffffff816b42a2>] od_dbs_timer+0xa2/0xf0
  [  288.421585]  [<ffffffff816b5fa4>] dbs_timer+0xb4/0xe0
  [  288.421589]  [<ffffffff8109a052>] process_one_work+0x162/0x480
  [  288.421592]  [<ffffffff8109a3bb>] worker_thread+0x4b/0x4c0
  [  288.421595]  [<ffffffff8109a370>] ? process_one_work+0x480/0x480
  [  288.421598]  [<ffffffff8109a370>] ? process_one_work+0x480/0x480
  [  288.421601]  [<ffffffff810a0588>] kthread+0xd8/0xf0
  [  288.421604]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
  [  288.421606]  [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
  [  288.421609]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
  [  288.421640] Code: 12 48 c1 e9 0c 83 e8 01 83 e1 30 48 98 48 81 c1 40 78 01 00 48 03 0c c5 80 42 f3 81 48 89 11 8b 42 08 85 c0 75 0d f3 90 8b 42 08 <85> c0 74 f7 eb 02 f3 90 8b 0f 66 85 c9 75 f7 39 f1 75 0f 89 c8 
  [  288.429453] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:0:20]
  [  288.429497] Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 8021q garp mrp stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip_tables x_tables openvswitch nf_defrag_ipv6 nf_conntrack nls_iso8859_1 ipmi_ssif intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass input_leds 8250_fintek sb_edac edac_core ipmi_si hpilo ipmi_msghandler lpc_ich ioatdma shpchp acpi_power_meter mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic crct10dif_pclmul crc32_pclmul usbhid hid ixgbe aesni_intel dca aes_x86_64 vxlan lrw gf128mul ip6_udp_tunnel tg3 glue_helper udp_tunnel hpsa ablk_helper ptp cryptd pps_core scsi_transport_sas mdio wmi fjes
  [  288.429508] CPU: 2 PID: 20 Comm: kworker/2:0 Tainted: G        W    L  4.4.0-22-generic #39-Ubuntu
  [  288.429509] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015
  [  288.429514] Workqueue: events dbs_timer
  [  288.429516] task: ffff881038466e00 ti: ffff881038044000 task.ti: ffff881038044000
  [  288.429521] RIP: 0010:[<ffffffff810ca84d>]  [<ffffffff810ca84d>] native_queued_spin_lock_slowpath+0x10d/0x170
  [  288.429523] RSP: 0018:ffff881038047c48  EFLAGS: 00000246
  [  288.429524] RAX: 0000000000000000 RBX: 000000000a52d280 RCX: ffff88103f717840
  [  288.429526] RDX: ffff88103f697840 RSI: 00000000000c0000 RDI: ffffffff821ed04c
  [  288.429527] RBP: ffff881038047c48 R08: 0000000000000101 R09: 0000000000000004
  [  288.429528] R10: ffff882033502c20 R11: ffff882033502c00 R12: ffffe8efffe84ef0
  [  288.429530] R13: ffff882033502c00 R14: 0000000000000002 R15: 0000000000000100
  [  288.429532] FS:  0000000000000000(0000) GS:ffff88103f680000(0000) knlGS:0000000000000000
  [  288.429533] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  [  288.429535] CR2: 0000000000604180 CR3: 0000000002e0a000 CR4: 00000000001406e0
  [  288.429536] Stack:
  [  288.429539]  ffff881038047c58 ffffffff81825061 ffff881038047cb0 ffffffff816b9d23
  [  288.429542]  ffff88103f696d00 0000000000000020 0018817000000100 ffff8802001a6da0
  [  288.429544]  0000000070d943b3 ffff882033502c00 00000000001a6da0 0000000000000002
  [  288.429545] Call Trace:
  [  288.429551]  [<ffffffff81825061>] _raw_spin_lock+0x21/0x30
  [  288.429553]  [<ffffffff816b9d23>] pcc_cpufreq_target+0x73/0x220
  [  288.429556]  [<ffffffff816afe38>] __cpufreq_driver_target+0x88/0x370
  [  288.429559]  [<ffffffff810fe28a>] ? get_cpu_idle_time_us+0x5a/0xd0
  [  288.429563]  [<ffffffff816b3fe1>] od_check_cpu+0xc1/0xd0
  [  288.429566]  [<ffffffff816b5d6e>] dbs_check_cpu+0x16e/0x1d0
  [  288.429569]  [<ffffffff816b42a2>] od_dbs_timer+0xa2/0xf0
  [  288.429572]  [<ffffffff816b5fa4>] dbs_timer+0xb4/0xe0
  [  288.429575]  [<ffffffff8109a052>] process_one_work+0x162/0x480
  [  288.429578]  [<ffffffff8109a3bb>] worker_thread+0x4b/0x4c0
  [  288.429581]  [<ffffffff8109a370>] ? process_one_work+0x480/0x480
  [  288.429583]  [<ffffffff810a0588>] kthread+0xd8/0xf0
  [  288.429586]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
  [  288.429588]  [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
  [  288.429590]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
  [  288.429620] Code: 12 48 c1 e9 0c 83 e8 01 83 e1 30 48 98 48 81 c1 40 78 01 00 48 03 0c c5 80 42 f3 81 48 89 11 8b 42 08 85 c0 75 0d f3 90 8b 42 08 <85> c0 74 f7 eb 02 f3 90 8b 0f 66 85 c9 75 f7 39 f1 75 0f 89 c8 
  ================================================================

  
  and another:

  ================================================================

  ovs1604 login: [ 1138.011941] BUG: unable to handle kernel NULL pointer dereference at 0000000000000400
  [ 1138.394597] IP: [<ffffffff81716136>] __netdev_pick_tx+0x66/0x140
  [ 1138.688303] PGD 0 
  [ 1138.785779] Oops: 0000 [#1] SMP 
  [ 1138.943399] Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 8021q garp mrp stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip_tables x_tables openvswitch nf_defrag_ipv6 nf_conntrack nls_iso8859_1 ipmi_ssif intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass input_leds sb_edac edac_core hpilo lpc_ich ipmi_si ipmi_msghandler 8250_fintek ioatdma shpchp acpi_power_meter mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic ixgbe crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 dca lrw vxlan gf128mul ip6_udp_tunnel glue_helper tg3 ablk_helper udp_tunnel cryptd hpsa usbhid ptp pps_core hid scsi_transport_sas mdio wmi fjes
  [ 1142.819687] CPU: 29 PID: 0 Comm: swapper/29 Not tainted 4.4.0-22-generic #39-Ubuntu
  [ 1143.193063] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015
  [ 1143.511272] task: ffff882038bc2940 ti: ffff881038404000 task.ti: ffff881038404000
  [ 1143.876349] RIP: 0010:[<ffffffff81716136>]  [<ffffffff81716136>] __netdev_pick_tx+0x66/0x140
  [ 1144.289553] RSP: 0018:ffff88103f9c36b0  EFLAGS: 00010206
  [ 1144.548342] RAX: ffff88202e15ec00 RBX: ffff88100dddd400 RCX: ffffffff817160d0
  [ 1144.898612] RDX: 000000000000007e RSI: ffff88100dddd400 RDI: ffff88102fda0000
  [ 1145.247504] RBP: ffff88103f9c36e0 R08: 0000000000000074 R09: ffff88100dbe2818
  [ 1145.595710] R10: ffff88100deba06c R11: 00000000000001b0 R12: ffff88102fda0000
  [ 1145.944483] R13: 0000000000000000 R14: 00000000ffffffff R15: 0000000000000400
  [ 1146.293274] FS:  0000000000000000(0000) GS:ffff88103f9c0000(0000) knlGS:0000000000000000
  [ 1146.688266] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  [ 1146.968958] CR2: 0000000000000400 CR3: 0000000002e0a000 CR4: 00000000001406e0
  [ 1147.322813] Stack:
  [ 1147.421064]  ffff881033968e00 ffff88100dddd400 ffff88102fda0000[ 1147.692532] ------------[ cut here ]------------
  [ 1147.692542] WARNING: CPU: 24 PID: 0 at /build/linux-UbQGH5/linux-4.4.0/net/sched/sch_generic.c:303 dev_watchdog+0x237/0x240()
  [ 1147.692543] NETDEV WATCHDOG: eth5 (ixgbe): transmit queue 13 timed out
  [ 1147.692544] Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 8021q garp mrp stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip_tables x_tables openvswitch nf_defrag_ipv6 nf_conntrack nls_iso8859_1 ipmi_ssif intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass input_leds sb_edac edac_core hpilo lpc_ich ipmi_si ipmi_msghandler 8250_fintek ioatdma shpchp acpi_power_meter mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic ixgbe crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 dca lrw vxlan gf128mul ip6_udp_tunnel glue_helper tg3 ablk_helper udp_tunnel cryptd hpsa usbhid ptp pps_core hid scsi_transport_sas mdio wmi fjes
  [ 1147.696361] CPU: 24 PID: 0 Comm: swapper/24 Not tainted 4.4.0-22-generic #39-Ubuntu
  [ 1147.696362] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015
  [ 1147.696364]  0000000000000286 f29cb0207d8e6510 ffff88203f083d98 ffffffff813e9c53
  [ 1147.696367]  ffff88203f083de0 ffffffff81d62a90 ffff88203f083dd0 ffffffff81080fb2
  [ 1147.696566]  000000000000000d ffff881030505f00 0000000000000018 ffff88102fda0000
  [ 1147.696758] Call Trace:
  [ 1147.696760]  <IRQ>  [<ffffffff813e9c53>] dump_stack+0x63/0x90
  [ 1147.697762]  [<ffffffff81080fb2>] warn_slowpath_common+0x82/0xc0
  [ 1147.697974]  [<ffffffff8108104c>] warn_slowpath_fmt+0x5c/0x80
  [ 1147.698185]  [<ffffffff81741df7>] dev_watchdog+0x237/0x240
  [ 1147.698188]  [<ffffffff81741bc0>] ? qdisc_rcu_free+0x40/0x40
  [ 1147.698193]  [<ffffffff810ec3b5>] call_timer_fn+0x35/0x120
  [ 1147.698195]  [<ffffffff81741bc0>] ? qdisc_rcu_free+0x40/0x40
  [ 1147.698199]  [<ffffffff810ecd6a>] run_timer_softirq+0x23a/0x2f0
  [ 1147.698203]  [<ffffffff810859f1>] __do_softirq+0x101/0x290
  [ 1147.698206]  [<ffffffff81085cf3>] irq_exit+0xa3/0xb0
  [ 1147.698213]  [<ffffffff81827da2>] smp_apic_timer_interrupt+0x42/0x50
  [ 1147.698216]  [<ffffffff81826062>] apic_timer_interrupt+0x82/0x90
  [ 1147.698217]  <EOI>  [<ffffffff816bc411>] ? cpuidle_enter_state+0x111/0x2b0
  [ 1147.698226]  [<ffffffff816bc5e7>] cpuidle_enter+0x17/0x20
  [ 1147.698233]  [<ffffffff810c3db2>] call_cpuidle+0x32/0x60
  [ 1147.698236]  [<ffffffff816bc5c3>] ? cpuidle_select+0x13/0x20
  [ 1147.698239]  [<ffffffff810c4070>] cpu_startup_entry+0x290/0x350
  [ 1147.698245]  [<ffffffff81051714>] start_secondary+0x154/0x190
  [ 1147.698247] ---[ end trace 640662c79e82e205 ]---
  [ 1147.698252] ixgbe 0000:05:00.1 eth5: initiating reset due to tx timeout
  [ 1147.698277] ixgbe 0000:05:00.1 eth5: Reset adapter
  [ 1148.258500] ixgbe 0000:05:00.1 eth5: detected SFP+: 5
  [ 1148.468518] ixgbe 0000:05:00.1 eth5: NIC Link is Up 10 Gbps, Flow Control: RX/TX
  [ 1161.520473]  ffff88102fda0000
  [ 1161.710452]  0000000000000000 ffffffff817160d0 ffff88103f9c3718 ffffffffc0457ce2
  [ 1162.090787]  0000000000000000 ffff88102fda0000 ffff88100dddd400 0000000000000000
  [ 1162.466909] Call Trace:
  [ 1162.616986]  <IRQ> 
  [ 1162.710449]  [<ffffffff817160d0>] ? __skb_tx_hash+0xa0/0xa0
  [ 1163.027595]  [<ffffffffc0457ce2>] ixgbe_select_queue+0xb2/0x130 [ixgbe]
  [ 1163.366984]  [<ffffffff8171d821>] netdev_pick_tx+0x61/0xe0
  [ 1163.658701]  [<ffffffff8171d967>] __dev_queue_xmit+0xc7/0x590
  [ 1163.955472]  [<ffffffff8171de40>] dev_queue_xmit+0x10/0x20
  [ 1164.239831]  [<ffffffffc03567e8>] vlan_dev_hard_start_xmit+0x98/0x120 [8021q]
  [ 1164.602400]  [<ffffffff8171d5d9>] dev_hard_start_xmit+0x249/0x3d0
  [ 1164.914965]  [<ffffffff8171ddc6>] __dev_queue_xmit+0x526/0x590
  [ 1165.218331]  [<ffffffff8171de40>] dev_queue_xmit+0x10/0x20
  [ 1165.499745]  [<ffffffffc0332cba>] ovs_vport_send+0x4a/0xc0 [openvswitch]
  [ 1165.839193]  [<ffffffffc03246a3>] do_output.isra.28+0x43/0x170 [openvswitch]
  [ 1166.196050]  [<ffffffffc0325a09>] do_execute_actions+0x729/0x8b0 [openvswitch]
  [ 1166.562138]  [<ffffffffc0325d73>] ovs_execute_actions+0x33/0xd0 [openvswitch]
  [ 1166.921220]  [<ffffffffc0329564>] ovs_dp_process_packet+0x84/0x130 [openvswitch]
  [ 1167.295247]  [<ffffffffc0325dee>] ovs_execute_actions+0xae/0xd0 [openvswitch]
  [ 1167.654452]  [<ffffffffc0329564>] ovs_dp_process_packet+0x84/0x130 [openvswitch]
  [ 1168.026460]  [<ffffffffc032a354>] ? key_extract+0x994/0xc80 [openvswitch]
  [ 1168.371007]  [<ffffffffc03325ac>] ovs_vport_receive+0x6c/0xd0 [openvswitch]
  [ 1168.722503]  [<ffffffff8171db26>] ? __dev_queue_xmit+0x286/0x590
  [ 1169.027910]  [<ffffffffc01d16c0>] ? nf_nat_ipv4_fn+0x60/0x220 [nf_nat_ipv4]
  [ 1169.378769]  [<ffffffff8175dd09>] ? ip_finish_output2+0x159/0x340
  [ 1169.688345]  [<ffffffffc02fd23d>] ? ipv4_confirm+0x7d/0x100 [nf_conntrack_ipv4]
  [ 1170.059626]  [<ffffffff8175ec56>] ? ip_finish_output+0x136/0x1f0
  [ 1170.364133]  [<ffffffff817524d3>] ? nf_hook_slow+0x73/0xd0
  [ 1170.639235]  [<ffffffff8175b2ed>] ? ip_forward_finish+0x4d/0x70
  [ 1170.939768]  [<ffffffffc03337cd>] netdev_frame_hook+0xed/0x160 [openvswitch]
  [ 1171.296146]  [<ffffffff8171af74>] __netif_receive_skb_core+0x364/0xa60
  [ 1171.626731]  [<ffffffff81794900>] ? inet_gro_receive+0x180/0x200
  [ 1171.931280]  [<ffffffff8171bf38>] ? dev_gro_receive+0x228/0x3c0
  [ 1172.241496]  [<ffffffff8171b688>] __netif_receive_skb+0x18/0x60
  [ 1172.541952]  [<ffffffff8171b702>] netif_receive_skb_internal+0x32/0xa0
  [ 1172.621141] NMI watchdog: BUG: soft lockup - CPU#14 stuck for 23s! [kworker/14:2:2021]
  [ 1172.621183] Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 8021q garp mrp stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip_tables x_tables openvswitch nf_defrag_ipv6 nf_conntrack nls_iso8859_1 ipmi_ssif intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass input_leds sb_edac edac_core hpilo lpc_ich ipmi_si ipmi_msghandler 8250_fintek ioatdma shpchp acpi_power_meter mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic ixgbe crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 dca lrw vxlan gf128mul ip6_udp_tunnel glue_helper tg3 ablk_helper udp_tunnel cryptd hpsa usbhid ptp pps_core hid scsi_transport_sas mdio wmi fjes
  [ 1172.621194] CPU: 14 PID: 2021 Comm: kworker/14:2 Tainted: G        W       4.4.0-22-generic #39-Ubuntu
  [ 1172.621195] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015
  [ 1172.621202] Workqueue: events linkwatch_event
  [ 1172.621204] task: ffff882032295280 ti: ffff882027830000 task.ti: ffff882027830000
  [ 1172.621215] RIP: 0010:[<ffffffff81103a2d>]  [<ffffffff81103a2d>] smp_call_function_single+0x8d/0x130
  [ 1172.621217] RSP: 0018:ffff882027833c40  EFLAGS: 00000202
  [ 1172.621218] RAX: 0000000000000001 RBX: 000000000000001f RCX: 0000000000000000
  [ 1172.621219] RDX: ffffffff81e52580 RSI: ffff88203ee17c00 RDI: 000000000000001f
  [ 1172.621220] RBP: ffff882027833c88 R08: 0000000000000000 R09: 0000000000000000
  [ 1172.621221] R10: 0000000000000004 R11: ffff8820322952e0 R12: ffffffff810e4d50
  [ 1172.621222] R13: 0000000080000000 R14: 00dffeff5ff26fd7 R15: 000000000000001f
  [ 1172.621224] FS:  0000000000000000(0000) GS:ffff88203ee00000(0000) knlGS:0000000000000000
  [ 1172.621225] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  [ 1172.621226] CR2: 00007f7c0be726d0 CR3: 0000000002e0a000 CR4: 00000000001406e0
  [ 1172.621227] Stack:
  [ 1172.621230]  ffff88103f007480 ffff88100ddc6600 0000000000000000 0000000000000000
  [ 1172.621232]  0000000000000000 0000000000000003 00000000b542f39f ffffffff81e52580
  [ 1172.621234]  000000000000dca0 ffff882027833d08 ffffffff810e53d9 ffffffff81e52700
  [ 1172.621234] Call Trace:
  [ 1172.621240]  [<ffffffff810e53d9>] synchronize_sched_expedited+0x339/0x880
  [ 1172.621244]  [<ffffffff810ec4f4>] ? lock_timer_base.isra.22+0x54/0x70
  [ 1172.621247]  [<ffffffff810e59ae>] synchronize_rcu_expedited+0xe/0x10
  [ 1172.621250]  [<ffffffff81716fd3>] synchronize_net+0x23/0x30
  [ 1172.621253]  [<ffffffff817429eb>] dev_deactivate_many+0x23b/0x240
  [ 1172.621255]  [<ffffffff81742a35>] dev_deactivate+0x45/0x70
  [ 1172.621258]  [<ffffffff817310d0>] linkwatch_do_dev+0x30/0x50
  [ 1172.621261]  [<ffffffff81731427>] __linkwatch_run_queue+0x167/0x190
  [ 1172.621263]  [<ffffffff81731475>] linkwatch_event+0x25/0x30
  [ 1172.621269]  [<ffffffff8109a052>] process_one_work+0x162/0x480
  [ 1172.621271]  [<ffffffff8109a3bb>] worker_thread+0x4b/0x4c0
  [ 1172.621274]  [<ffffffff8109a370>] ? process_one_work+0x480/0x480
  [ 1172.621277]  [<ffffffff8109a370>] ? process_one_work+0x480/0x480
  [ 1172.621280]  [<ffffffff810a0588>] kthread+0xd8/0xf0
  [ 1172.621283]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
  [ 1172.621285]  [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
  [ 1172.621287]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
  [ 1172.621309] Code: c0 74 0c 9c 58 0f 1f 44 00 00 f6 c4 02 74 71 85 c9 75 48 48 c7 c6 00 7c 01 00 65 48 03 35 64 67 f0 7e 8b 46 18 a8 01 74 09 f3 90 <8b> 46 18 a8 01 75 f7 83 4e 18 01 48 89 d1 89 df 4c 89 e2 e8 3b 
  [ 1189.966808]  [<ffffffff8171b85f>] napi_gro_complete+0x7f/0xa0
  [ 1190.263283]  [<ffffffff8171b8da>] napi_gro_flush+0x5a/0x80
  [ 1190.546867]  [<ffffffffc045a32e>] ixgbe_poll+0x58e/0x8d0 [ixgbe]
  [ 1190.855476]  [<ffffffff8171bbce>] net_rx_action+0x21e/0x360
  [ 1191.142638]  [<ffffffff810859f1>] __do_softirq+0x101/0x290
  [ 1191.424157]  [<ffffffff81085cf3>] irq_exit+0xa3/0xb0
  [ 1191.679965]  [<ffffffff81827ce4>] do_IRQ+0x54/0xd0
  [ 1191.926377]  [<ffffffff81825dc2>] common_interrupt+0x82/0x82
  [ 1192.228819]  <EOI> 
  [ 1192.322267]  [<ffffffff816bc411>] ? cpuidle_enter_state+0x111/0x2b0
  [ 1192.660739]  [<ffffffff816bc5e7>] cpuidle_enter+0x17/0x20
  [ 1192.943489]  [<ffffffff810c3db2>] call_cpuidle+0x32/0x60
  [ 1193.215091]  [<ffffffff816bc5c3>] ? cpuidle_select+0x13/0x20
  [ 1193.499665]  [<ffffffff810c4070>] cpu_startup_entry+0x290/0x350
  [ 1193.801670]  [<ffffffff81051714>] start_secondary+0x154/0x190
  [ 1194.092982] Code: 87 e8 03 00 00 49 89 fc 48 85 c0 0f 84 da 00 00 00 8b 96 ac 00 00 00 83 ea 01 48 8d 44 d0 10 4c 8b 38 4d 85 ff 0f 84 c0 00 00 00 <41> 8b 1f 83 fb 01 0f 84 8d 00 00 00 f6 86 91 00 00 00 30 0f 84 
  [ 1195.043812] RIP  [<ffffffff81716136>] __netdev_pick_tx+0x66/0x140
  [ 1195.352932]  RSP <ffff88103f9c36b0>
  [ 1195.534578] CR2: 0000000000000400
  [ 1195.742792] ---[ end trace 640662c79e82e206 ]---
  [ 1195.981707] Kernel panic - not syncing: Fatal exception in interrupt
  [ 1196.303793] Kernel Offset: disabled
  [ 1196.493128] ---[ end Kernel panic - not syncing: Fatal exception in interrupt
  [ 1196.854462] ------------[ cut here ]------------
  [ 1197.090915] WARNING: CPU: 29 PID: 0 at /build/linux-UbQGH5/linux-4.4.0/arch/x86/kernel/smp.c:125 native_smp_send_reschedule+0x60/0x70()
  [ 1197.706261] Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 8021q garp mrp stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip_tables x_tables openvswitch nf_defrag_ipv6 nf_conntrack nls_iso8859_1 ipmi_ssif intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass input_leds sb_edac edac_core hpilo lpc_ich ipmi_si ipmi_msghandler 8250_fintek ioatdma shpchp acpi_power_meter mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic ixgbe crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 dca lrw vxlan gf128mul ip6_udp_tunnel glue_helper tg3 ablk_helper udp_tunnel cryptd hpsa usbhid ptp pps_core hid scsi_transport_sas mdio wmi fjes
  [ 1201.718155] CPU: 29 PID: 0 Comm: swapper/29 Tainted: G      D W    L  4.4.0-22-generic #39-Ubuntu
  [ 1202.164963] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015
  [ 1202.499892]  0000000000000086 f9c2da801aceb89c ffff88103f9c3168 ffffffff813e9c53
  [ 1202.880066]  0000000000000000 ffffffff81ca35b0 ffff88103f9c31a0 ffffffff81080fb2
  [ 1203.259124]  000000000000000f ffff88203ee56d00 000000000000001d ffff882038bc2940
  [ 1203.636609] Call Trace:
  [ 1203.773175]  <IRQ>  [<ffffffff813e9c53>] dump_stack+0x63/0x90
  [ 1204.070769]  [<ffffffff81080fb2>] warn_slowpath_common+0x82/0xc0
  [ 1204.383768]  [<ffffffff810810fa>] warn_slowpath_null+0x1a/0x20
  [ 1204.685193]  [<ffffffff81050970>] native_smp_send_reschedule+0x60/0x70
  [ 1205.020826]  [<ffffffff810be7a3>] trigger_load_balance+0x133/0x210
  [ 1205.334824]  [<ffffffff810ad3a6>] scheduler_tick+0xa6/0xd0
  [ 1205.619034]  [<ffffffff810fe560>] ? tick_sched_handle.isra.14+0x60/0x60
  [ 1205.959021]  [<ffffffff810ee961>] update_process_times+0x51/0x60
  [ 1206.270252]  [<ffffffff810fe525>] tick_sched_handle.isra.14+0x25/0x60
  [ 1206.601615]  [<ffffffff810fe59d>] tick_sched_timer+0x3d/0x70
  [ 1206.894935]  [<ffffffff810ef282>] __hrtimer_run_queues+0x102/0x290
  [ 1207.209571]  [<ffffffff810efa48>] hrtimer_interrupt+0xa8/0x1a0
  [ 1207.510638]  [<ffffffff81052fa8>] local_apic_timer_interrupt+0x38/0x60
  [ 1207.845707]  [<ffffffff81827d9d>] smp_apic_timer_interrupt+0x3d/0x50
  [ 1208.172391]  [<ffffffff81826062>] apic_timer_interrupt+0x82/0x90
  [ 1208.482226]  [<ffffffff8118bbb9>] ? panic+0x1d5/0x215
  [ 1208.745324]  [<ffffffff8118bbb2>] ? panic+0x1ce/0x215
  [ 1209.008398]  [<ffffffff81031c4a>] oops_end+0xca/0xd0
  [ 1209.263560]  [<ffffffff8106ad45>] no_context+0x135/0x380
  [ 1209.538824]  [<ffffffff8106b010>] __bad_area_nosemaphore+0x80/0x1f0
  [ 1209.860782]  [<ffffffff8106b193>] bad_area_nosemaphore+0x13/0x20
  [ 1210.170022]  [<ffffffff8106b457>] __do_page_fault+0xb7/0x400
  [ 1210.463123]  [<ffffffffc0462a02>] ? ixgbe_xmit_frame+0x42/0x90 [ixgbe]
  [ 1210.797385]  [<ffffffff8106b7c2>] do_page_fault+0x22/0x30
  [ 1211.076463]  [<ffffffff81827478>] page_fault+0x28/0x30
  [ 1211.342835]  [<ffffffff817160d0>] ? __skb_tx_hash+0xa0/0xa0
  [ 1211.633287]  [<ffffffff81716136>] ? __netdev_pick_tx+0x66/0x140
  [ 1211.940942]  [<ffffffff817160d0>] ? __skb_tx_hash+0xa0/0xa0
  [ 1212.230203]  [<ffffffffc0457ce2>] ixgbe_select_queue+0xb2/0x130 [ixgbe]
  [ 1212.566347]  [<ffffffff8171d821>] netdev_pick_tx+0x61/0xe0
  [ 1212.851473]  [<ffffffff8171d967>] __dev_queue_xmit+0xc7/0x590
  [ 1213.152225]  [<ffffffff8171de40>] dev_queue_xmit+0x10/0x20
  [ 1213.437482]  [<ffffffffc03567e8>] vlan_dev_hard_start_xmit+0x98/0x120 [8021q]
  [ 1213.801585]  [<ffffffff8171d5d9>] dev_hard_start_xmit+0x249/0x3d0
  [ 1214.115849]  [<ffffffff8171ddc6>] __dev_queue_xmit+0x526/0x590
  [ 1214.416730]  [<ffffffff8171de40>] dev_queue_xmit+0x10/0x20
  [ 1214.699175]  [<ffffffffc0332cba>] ovs_vport_send+0x4a/0xc0 [openvswitch]
  [ 1215.038336]  [<ffffffffc03246a3>] do_output.isra.28+0x43/0x170 [openvswitch]
  [ 1215.394550]  [<ffffffffc0325a09>] do_execute_actions+0x729/0x8b0 [openvswitch]
  [ 1215.758636]  [<ffffffffc0325d73>] ovs_execute_actions+0x33/0xd0 [openvswitch]
  [ 1216.118188]  [<ffffffffc0329564>] ovs_dp_process_packet+0x84/0x130 [openvswitch]
  [ 1216.490791]  [<ffffffffc0325dee>] ovs_execute_actions+0xae/0xd0 [openvswitch]
  [ 1216.849706]  [<ffffffffc0329564>] ovs_dp_process_packet+0x84/0x130 [openvswitch]
  [ 1217.222344]  [<ffffffffc032a354>] ? key_extract+0x994/0xc80 [openvswitch]
  [ 1217.566366]  [<ffffffffc03325ac>] ovs_vport_receive+0x6c/0xd0 [openvswitch]
  [ 1217.918259]  [<ffffffff8171db26>] ? __dev_queue_xmit+0x286/0x590
  [ 1218.222395]  [<ffffffffc01d16c0>] ? nf_nat_ipv4_fn+0x60/0x220 [nf_nat_ipv4]
  [ 1218.572934]  [<ffffffff8175dd09>] ? ip_finish_output2+0x159/0x340
  [ 1218.881368]  [<ffffffffc02fd23d>] ? ipv4_confirm+0x7d/0x100 [nf_conntrack_ipv4]
  [ 1219.248697]  [<ffffffff8175ec56>] ? ip_finish_output+0x136/0x1f0
  [ 1219.552761]  [<ffffffff817524d3>] ? nf_hook_slow+0x73/0xd0
  [ 1219.832000]  [<ffffffff8175b2ed>] ? ip_forward_finish+0x4d/0x70
  [ 1220.131812]  [<ffffffffc03337cd>] netdev_frame_hook+0xed/0x160 [openvswitch]
  [ 1220.486445]  [<ffffffff8171af74>] __netif_receive_skb_core+0x364/0xa60
  [ 1220.816076]  [<ffffffff81794900>] ? inet_gro_receive+0x180/0x200
  [ 1221.119917]  [<ffffffff8171bf38>] ? dev_gro_receive+0x228/0x3c0
  [ 1221.419505]  [<ffffffff8171b688>] __netif_receive_skb+0x18/0x60
  [ 1221.718992]  [<ffffffff8171b702>] netif_receive_skb_internal+0x32/0xa0
  [ 1222.048204]  [<ffffffff8171b85f>] napi_gro_complete+0x7f/0xa0
  [ 1222.330734]  [<ffffffff8171b8da>] napi_gro_flush+0x5a/0x80
  [ 1222.609174]  [<ffffffffc045a32e>] ixgbe_poll+0x58e/0x8d0 [ixgbe]
  [ 1222.914199]  [<ffffffff8171bbce>] net_rx_action+0x21e/0x360
  [ 1223.196818]  [<ffffffff810859f1>] __do_softirq+0x101/0x290
  [ 1223.475092]  [<ffffffff81085cf3>] irq_exit+0xa3/0xb0
  [ 1223.728540]  [<ffffffff81827ce4>] do_IRQ+0x54/0xd0
  [ 1223.973680]  [<ffffffff81825dc2>] common_interrupt+0x82/0x82
  [ 1224.260746]  <EOI>  [<ffffffff816bc411>] ? cpuidle_enter_state+0x111/0x2b0
  [ 1224.607877]  [<ffffffff816bc5e7>] cpuidle_enter+0x17/0x20
  [ 1224.886873]  [<ffffffff810c3db2>] call_cpuidle+0x32/0x60
  [ 1225.158221]  [<ffffffff816bc5c3>] ? cpuidle_select+0x13/0x20
  [ 1225.447849]  [<ffffffff810c4070>] cpu_startup_entry+0x290/0x350
  [ 1225.748616]  [<ffffffff81051714>] start_secondary+0x154/0x190
  [ 1226.047825] ---[ end trace 640662c79e82e207 ]---

  =======================================================

  
  I'm not sure if providing the current openvswitch configuration will help.  The setup is fairly complicated.  Here is the basic configuration:

  # ovs-vsctl show
  1a7ea432-8b96-445e-8734-d71d10533302
      Bridge "ovsbr0"
          Port "eth6"
              tag: 998
              Interface "eth6"
          Port "eth5.110"
              tag: 110
              Interface "eth5.110"
          Port "eth4"
              tag: 999
              Interface "eth4"
          Port int-servers
              tag: 998
              Interface int-servers
                  type: internal
          Port client
              tag: 999
              Interface client
                  type: internal
          Port "ovsbr0"
              Interface "ovsbr0"
                  type: internal
          Port ingress
              tag: 110
              Interface ingress
                  type: internal
          Port "eth1.110"
              tag: 110
              Interface "eth1.110"
      ovs_version: "2.5.0"

  
  # ovs-ofctl -O Openflow15 dump-groups ovsbr0
  OFPST_GROUP_DESC reply (OF1.5) (xid=0x2):
   group_id=200,type=select,selection_method=hash,fields(ip_dst,tcp_dst),bucket=bucket_id:0,actions=set_field:fa:16:3e:17:2f:f1->eth_dst,output:9,bucket=bucket_id:1,actions=set_field:fa:16:3e:2f:51:99->eth_dst,output:9,bucket=bucket_id:2,actions=set_field:fa:16:3e:76:f3:42->eth_dst,output:9,bucket=bucket_id:3,actions=set_field:fa:16:3e:c0:cf:10->eth_dst,output:9,bucket=bucket_id:4,actions=set_field:fa:16:3e:ba:6f:b7->eth_dst,output:9
   group_id=100,type=select,selection_method=hash,fields(ip_src,tcp_src),bucket=bucket_id:0,actions=set_field:fa:16:3e:17:2f:f1->eth_dst,output:9,bucket=bucket_id:1,actions=set_field:fa:16:3e:2f:51:99->eth_dst,output:9,bucket=bucket_id:2,actions=set_field:fa:16:3e:76:f3:42->eth_dst,output:9,bucket=bucket_id:3,actions=set_field:fa:16:3e:c0:cf:10->eth_dst,output:9,bucket=bucket_id:4,actions=set_field:fa:16:3e:ba:6f:b7->eth_dst,output:9

  # ovs-ofctl -O Openflow15 dump-flows ovsbr0
  OFPST_FLOW reply (OF1.5) (xid=0x2):
   cookie=0x0, duration=206.013s, table=0, n_packets=0, n_bytes=0, priority=100,in_port=2 actions=resubmit(,100)
   cookie=0x0, duration=206.013s, table=0, n_packets=0, n_bytes=0, priority=100,in_port=10 actions=resubmit(,101)
   cookie=0x0, duration=206.009s, table=0, n_packets=8, n_bytes=632, priority=100,in_port=9 actions=resubmit(,200)
   cookie=0x0, duration=206.013s, table=0, n_packets=8, n_bytes=560, priority=0 actions=NORMAL
   cookie=0x0, duration=206.012s, table=100, n_packets=0, n_bytes=0, priority=100,ct_state=-trk,tcp,nw_src=10.100.0.0/16,tp_dst=80 actions=ct(table=110)
   cookie=0x0, duration=206.012s, table=100, n_packets=0, n_bytes=0, priority=0 actions=NORMAL
   cookie=0x0, duration=206.012s, table=101, n_packets=0, n_bytes=0, priority=100,ct_state=-trk,tcp,nw_dst=10.100.0.0/16,tp_src=80 actions=ct(table=120,zone=200)
   cookie=0x0, duration=206.012s, table=101, n_packets=0, n_bytes=0, priority=0 actions=NORMAL
   cookie=0x0, duration=206.011s, table=110, n_packets=0, n_bytes=0, priority=100,ct_state=+new+trk,ip actions=ct(commit,exec(set_field:0x64->ct_mark)),group:100
   cookie=0x0, duration=206.010s, table=110, n_packets=0, n_bytes=0, priority=100,ct_state=+est-rpl+trk,ip actions=ct(commit),group:100
   cookie=0x0, duration=206.010s, table=110, n_packets=0, n_bytes=0, priority=0 actions=drop
   cookie=0x0, duration=206.010s, table=120, n_packets=0, n_bytes=0, priority=101,ct_state=+est+rpl+trk,ct_zone=200,ip actions=ct(commit,zone=200),group:200
   cookie=0x0, duration=206.010s, table=120, n_packets=0, n_bytes=0, priority=100,ct_state=+est+rpl+trk,ip actions=ct(commit,zone=200),group:200
   cookie=0x0, duration=206.009s, table=120, n_packets=0, n_bytes=0, priority=0 actions=drop
   cookie=0x0, duration=206.009s, table=200, n_packets=0, n_bytes=0, priority=100,ct_state=-trk,tcp,nw_src=10.100.0.0/16,tp_dst=80 actions=ct(table=210,zone=200)
   cookie=0x0, duration=206.008s, table=200, n_packets=0, n_bytes=0, priority=100,ct_state=-trk,tcp,nw_dst=10.100.0.0/16,tp_src=80 actions=ct(table=220)
   cookie=0x0, duration=206.008s, table=200, n_packets=8, n_bytes=632, priority=0 actions=NORMAL
   cookie=0x0, duration=206.008s, table=210, n_packets=0, n_bytes=0, priority=100,ct_state=+new+trk,ip actions=ct(commit,zone=200),NORMAL
   cookie=0x0, duration=206.007s, table=210, n_packets=0, n_bytes=0, priority=100,ct_state=+est-rpl+trk,ip actions=ct(commit,zone=200),NORMAL
   cookie=0x0, duration=206.007s, table=210, n_packets=0, n_bytes=0, priority=0 actions=drop
   cookie=0x0, duration=206.006s, table=220, n_packets=0, n_bytes=0, priority=100,ct_state=+est+rpl+trk,ip actions=ct(commit),NORMAL
   cookie=0x0, duration=206.006s, table=220, n_packets=0, n_bytes=0, priority=100,ct_state=+rpl+trk,ip actions=ct(commit),NORMAL
   cookie=0x0, duration=206.006s, table=220, n_packets=0, n_bytes=0, priority=0 actions=drop

  ProblemType: Bug
  DistroRelease: Ubuntu 16.04
  Package: linux-image-4.4.0-22-generic 4.4.0-22.39
  ProcVersionSignature: Ubuntu 4.4.0-22.39-generic 4.4.8
  Uname: Linux 4.4.0-22-generic x86_64
  AlsaDevices:
   total 0
   crw-rw---- 1 root audio 116,  1 May  9 17:05 seq
   crw-rw---- 1 root audio 116, 33 May  9 17:05 timer
  AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
  ApportVersion: 2.20.1-0ubuntu2
  Architecture: amd64
  ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
  AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
  Date: Mon May  9 17:12:43 2016
  HibernationDevice: RESUME=/dev/mapper/ovs1604--vg-swap
  InstallationDate: Installed on 2016-05-09 (0 days ago)
  InstallationMedia: Ubuntu-Server 16.04 LTS "Xenial Xerus" - Release amd64 (20160420.3)
  IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
  MachineType: HP ProLiant DL380 Gen9
  PciMultimedia:
   
  ProcFB: 0 EFI VGA
  ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-22-generic.efi.signed root=/dev/mapper/hostname--vg-root ro net.ifnames=0 biosdevname=0 console=ttyS0,9600 console=tty1
  RelatedPackageVersions:
   linux-restricted-modules-4.4.0-22-generic N/A
   linux-backports-modules-4.4.0-22-generic  N/A
   linux-firmware                            1.157
  RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
  SourcePackage: linux
  UpgradeStatus: No upgrade log present (probably fresh install)
  dmi.bios.date: 05/06/2015
  dmi.bios.vendor: HP
  dmi.bios.version: P89
  dmi.chassis.type: 23
  dmi.chassis.vendor: HP
  dmi.modalias: dmi:bvnHP:bvrP89:bd05/06/2015:svnHP:pnProLiantDL380Gen9:pvr:cvnHP:ct23:cvr:
  dmi.product.name: ProLiant DL380 Gen9
  dmi.sys.vendor: HP

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1579943/+subscriptions


Follow ups