kernel-packages team mailing list archive

Thread
Date
[Bug 1152580] Re: [Dell PowerEdge M520] BCM95720 Network disconnects and resets

To: kernel-packages@xxxxxxxxxxxxxxxxxxx
From: wonko <bernard@xxxxxxxxxxxx>
Date: Wed, 18 Jun 2014 10:45:35 -0000
Reply-to: Bug 1152580 <1152580@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Hi,

we have a similar, or the same, problem with Dell PE machines, having
the Broadcom chip mentioned above. We have this problem on generation 12
systems, across different models (R420, R620), with several combinations
of bios firmwares, lifecycle firmwares, etc... We see this on several
versions of the linux kernel, ranging from 3.2.x up tot 3.11, with
several versions of the tg3 driver, including a manually compiled latest
version (3.133d) loaded in a 3.11. The latest machine, where we can
reproduce the problem has Ubuntu Precise installed, but we also see this
behavior on Debian machines. We run Xen on it, running HVM hosts on it.
Storage is handled over iSCSI (and it is the iSCSI interface we can
trigger this bug on in a reproducible way, while we have the impression
it also happens on other interfaces, but there we don't have a solid
case where we have e reproducible setup.

All this info actually points into the direction of the tg3 driver and
hardware below it not handling certain datastreams or data patterns
correctly, and finally crashing the system. It seems unrelated to the
version of kernel running, xen-version running, amount of VM's running,
firmwares and revisions running, etc...

We have been trying to pinpoint this for over a year now, being unable
to actually create a scenario where we could reproduce this. As of this
week, we finally found a specific setup where we could trigger the
error.

The error is triggered by running a certain VM, and inside that VM,
importing a mysqldump in a running mysql on that VM. The VM has it's
traffic on an iSCSI volume, so this effectually generates a datastream
over the eth1 interface of the machine.  Within a short amount of time,
the system will crash in 2 steps. We first see a timeout on the tg3
driver, and finally, the machine freezes and reboots.

Kernel boot:

[    0.000000] Linux version 3.11.0-19-generic (buildd@allspice) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #33~precise1-Ubuntu SMP Wed Mar 12 21:16:27 UTC 2014 (Ubuntu 3.11.0-19.33~precise1-generic 3.11.10.5)
[    0.000000] Command line: placeholder root=UUID=bbc71780-90bf-4647-b579-e48d5d8c2bce ro vga=0x317 crashkernel=348M-2G:64M,2G-:128M intremap=off
...

The intremap=off was a guess it had something to do with the interrupts
being handled wrongly, but this didn't matter.

Our tg3 hardware/driver init:

[   64.596574] tg3.c:v3.133d (August 07, 2013)
...
[   64.609739] pcieport 0000:00:1c.4: eth0: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address f0:1f:af:e9:33:da
[   64.609905] pcieport 0000:00:1c.4: eth0: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
[   64.610071] pcieport 0000:00:1c.4: eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
[   64.610244] pcieport 0000:00:1c.4: eth0: dma_rwctrl[00000001] dma_mask[64-bit]
...
[   64.621595] pcieport 0000:00:1c.4: eth1: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address f0:1f:af:e9:33:db
[   64.621602] pcieport 0000:00:1c.4: eth1: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
[   64.621605] pcieport 0000:00:1c.4: eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
[   64.621606] pcieport 0000:00:1c.4: eth1: dma_rwctrl[00000001] dma_mask[64-bit]
...
same for eth2 and eth3. Only eth0 and eth1 are in use, eth2 and 3 have no cable plugged in.

When generating the traffic, eventually this will show up:

[ 1053.598372]  connection1:0: detected conn error (1020)
[ 1059.389764] ------------[ cut here ]------------
[ 1059.389795] WARNING: CPU: 0 PID: 0 at /build/buildd/linux-lts-saucy-3.11.0/net/sched/sch_generic.c:260 dev_watchdog+0x262/0x270()
[ 1059.389799] NETDEV WATCHDOG: eth1 (tg3): transmit queue 0 timed out
[ 1059.389802] Modules linked in: xt_physdev xen_gntdev xen_netback xen_blkback xen_evtchn xenfs xen_privcmd mpt3sas mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_si ipmi_devintf ipmi_msghandler dell_rbu microcode ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi xt_tcpudp nf_conntrack_ipv6 nf_defrag_ipv6 xt_state ip6table_filter ip6_tables iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables bridge joydev hid_generic 8021q mrp usbhid garp stp llc gpio_ich dcdbas hid usb_storage shpchp mei_me mac_hid lpc_ich mei wmi acpi_power_meter parport_pc ppdev lp parport ahci libahci tg3(OF) ptp pps_core [last unloaded: ipmi_si]
[ 1059.389921] CPU: 0 PID: 0 Comm: swapper/0 Tainted: GF          O 3.11.0-19-generic #33~precise1-Ubuntu
[ 1059.389928] Hardware name: Dell Inc. PowerEdge R420/0JD6X3, BIOS 2.1.2 01/20/2014
[ 1059.389936]  0000000000000104 ffff880130e03cf0 ffffffff8173d82d 0000000000000000
[ 1059.389946]  ffff880130e03d40 ffff880130e03d30 ffffffff8106540c ffffffff8137d8f4
[ 1059.389953]  ffff880110db0000 0000000000000000 0000000000000005 0000000000000000
[ 1059.389961] Call Trace:
[ 1059.389965]  <IRQ>  [<ffffffff8173d82d>] dump_stack+0x46/0x58
[ 1059.389992]  [<ffffffff8106540c>] warn_slowpath_common+0x8c/0xc0
[ 1059.390005]  [<ffffffff8137d8f4>] ? timerqueue_add+0x64/0xb0
[ 1059.390011]  [<ffffffff810654f6>] warn_slowpath_fmt+0x46/0x50
[ 1059.390023]  [<ffffffff8165a5a2>] dev_watchdog+0x262/0x270
[ 1059.390033]  [<ffffffff8147ce46>] ? credit_entropy_bits.part.7+0x176/0x1d0
[ 1059.390046]  [<ffffffff8107f7a0>] ? __queue_work+0x3d0/0x3d0
[ 1059.390052]  [<ffffffff8165a340>] ? pfifo_fast_dequeue+0xe0/0xe0
[ 1059.390063]  [<ffffffff81072656>] call_timer_fn+0x46/0x160
[ 1059.390069]  [<ffffffff8147ceb2>] ? credit_entropy_bits+0x12/0x20
[ 1059.390079]  [<ffffffff81073987>] run_timer_softirq+0x267/0x2c0
[ 1059.390086]  [<ffffffff8165a340>] ? pfifo_fast_dequeue+0xe0/0xe0
[ 1059.390099]  [<ffffffff810ffbae>] ? irq_get_irq_data+0xe/0x10
[ 1059.390107]  [<ffffffff8106a5e0>] __do_softirq+0xe0/0x280
[ 1059.390124]  [<ffffffff810c59bd>] ? tick_do_update_jiffies64+0x9d/0xd0
[ 1059.390131]  [<ffffffff81753e1c>] call_softirq+0x1c/0x30
[ 1059.390142]  [<ffffffff81015e35>] do_softirq+0x65/0xa0
[ 1059.390154]  [<ffffffff8106a8fe>] irq_exit+0x9e/0xc0
[ 1059.390164]  [<ffffffff81438405>] xen_evtchn_do_upcall+0x35/0x50
[ 1059.390170]  [<ffffffff81753e7e>] xen_do_hypervisor_callback+0x1e/0x30
[ 1059.390172]  <EOI>  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[ 1059.390185]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[ 1059.390199]  [<ffffffff8100acf0>] ? xen_safe_halt+0x10/0x20
[ 1059.390207]  [<ffffffff8101c873>] ? default_idle+0x23/0xe0
[ 1059.390213]  [<ffffffff8101d106>] ? arch_cpu_idle+0x26/0x30
[ 1059.390222]  [<ffffffff810baae8>] ? cpu_idle_loop+0x98/0x260
[ 1059.390234]  [<ffffffff810bad1b>] ? cpu_startup_entry+0x6b/0x70
[ 1059.390243]  [<ffffffff81725787>] ? rest_init+0x77/0x80
[ 1059.390251]  [<ffffffff81d26f23>] ? start_kernel+0x407/0x414
[ 1059.390256]  [<ffffffff81d269ae>] ? repair_env_string+0x5a/0x5a
[ 1059.390262]  [<ffffffff81d265e8>] ? x86_64_start_reservations+0x2a/0x2c
[ 1059.390269]  [<ffffffff81d2a454>] ? xen_start_kernel+0x49a/0x49c
[ 1059.390273] ---[ end trace 2e073eeeb740ebd8 ]---
[ 1059.390278] pcieport 0000:00:1c.4: eth1: transmit timed out, resetting
[ 1060.656169] pcieport 0000:00:1c.4: eth1: 0x00000000: 0x165f14e4, 0x20100406, 0x02000000, 0x00800010
[ 1060.656296] pcieport 0000:00:1c.4: eth1: 0x00000010: 0xd91d000c, 0x00000000, 0xd91e000c, 0x00000000
[ 1060.656408] pcieport 0000:00:1c.4: eth1: 0x00000020: 0xd91f000c, 0x00000000, 0x00000000, 0x04f81028
[ 1060.656520] pcieport 0000:00:1c.4: eth1: 0x00000030: 0xdd800000, 0x00000048, 0x00000000, 0x0000020e
[ 1060.656631] pcieport 0000:00:1c.4: eth1: 0x00000040: 0x00000000, 0x10000000, 0xc8035001, 0x64002008
[ 1060.656741] pcieport 0000:00:1c.4: eth1: 0x00000050: 0x818c5803, 0x78000000, 0x0086a005, 0x00000000
[ 1060.656853] pcieport 0000:00:1c.4: eth1: 0x00000060: 0x00000000, 0x00000000, 0xf0000298, 0x00380081
[ 1060.656966] pcieport 0000:00:1c.4: eth1: 0x00000070: 0x000710b0, 0xfffc4740, 0x0001421c, 0x00000000
[ 1060.657102] pcieport 0000:00:1c.4: eth1: 0x00000080: 0x00000001, 0x40000040, 0x00000000, 0x00000024
[ 1060.657260] pcieport 0000:00:1c.4: eth1: 0x00000090: 0x00000000, 0x00000000, 0x00000000, 0x0000002e
[ 1060.657419] pcieport 0000:00:1c.4: eth1: 0x000000a0: 0x8010ac11, 0x00000004, 0x00001004, 0x00020010
[ 1060.657574] pcieport 0000:00:1c.4: eth1: 0x000000b0: 0x10008d81, 0x0010240e, 0x0004cc22, 0x10120040
[ 1060.657745] pcieport 0000:00:1c.4: eth1: 0x000000d0: 0x0000001f, 0x00000006, 0x00000000, 0x00000000
[ 1060.657906] pcieport 0000:00:1c.4: eth1: 0x000000f0: 0x00000000, 0x05720000, 0x00000000, 0x00000000
[ 1060.658064] pcieport 0000:00:1c.4: eth1: 0x00000100: 0x13c10001, 0x00000000, 0x00018000, 0x000e7030
[ 1060.658222] pcieport 0000:00:1c.4: eth1: 0x00000110: 0x00002000, 0x000031c0, 0x000000a0, 0x00000000
[ 1060.658376] pcieport 0000:00:1c.4: eth1: 0x00000130: 0x00000000, 0x00000000, 0x00000000, 0x15010003
[ 1060.658534] pcieport 0000:00:1c.4: eth1: 0x00000140: 0xafe933db, 0x0000f01f, 0x00000000, 0x00000000
[ 1060.658691] pcieport 0000:00:1c.4: eth1: 0x00000150: 0x16010004, 0x00000000, 0x0007811b, 0x00000001
[ 1060.658848] pcieport 0000:00:1c.4: eth1: 0x00000160: 0x00010002, 0x00000000, 0x00000000, 0x00000000
[ 1060.659001] pcieport 0000:00:1c.4: eth1: 0x00000170: 0x00000000, 0x800000ff, 0x00000000, 0x00000000
[ 1060.659158] pcieport 0000:00:1c.4: eth1: 0x00000200: 0x00000000, 0x10000000, 0x00000000, 0x25000000
[ 1060.659316] pcieport 0000:00:1c.4: eth1: 0x00000210: 0x00000000, 0x64000000, 0x00000000, 0x01000000
[ 1060.659473] pcieport 0000:00:1c.4: eth1: 0x00000220: 0x00000000, 0x08000000, 0x00000000, 0x00000000
[ 1060.659630] pcieport 0000:00:1c.4: eth1: 0x00000260: 0x00000000, 0x00000000, 0x00000000, 0x0000002e
[ 1060.659788] pcieport 0000:00:1c.4: eth1: 0x00000280: 0x00000000, 0x00000024, 0x00000000, 0x0000073a
[ 1060.659946] pcieport 0000:00:1c.4: eth1: 0x00000290: 0x00000000, 0x00000000, 0x00000000, 0x00000008
[ 1060.660103] pcieport 0000:00:1c.4: eth1: 0x00000300: 0x00000000, 0x0000004b, 0x00000000, 0x00000000
[ 1060.660262] pcieport 0000:00:1c.4: eth1: 0x00000400: 0x18e04808, 0x00400000, 0x00001000, 0x00000880
[ 1060.660420] pcieport 0000:00:1c.4: eth1: 0x00000410: 0x0000f01f, 0xafe933db, 0x0000f01f, 0xafe933db
[ 1060.660578] pcieport 0000:00:1c.4: eth1: 0x00000420: 0x0000f01f, 0xafe933db, 0x0000f01f, 0xafe933db
[ 1060.660735] pcieport 0000:00:1c.4: eth1: 0x00000430: 0x00000400, 0x00000000, 0x0000023f, 0x000005f2
[ 1060.660893] pcieport 0000:00:1c.4: eth1: 0x00000440: 0x00000000, 0x00000000, 0x00000000, 0x04584400
[ 1060.661051] pcieport 0000:00:1c.4: eth1: 0x00000450: 0x00000001, 0x00008000, 0x00000000, 0x00000102
[ 1060.661208] pcieport 0000:00:1c.4: eth1: 0x00000460: 0x00000008, 0x00002620, 0x01ff0002, 0x00000000
[ 1060.661366] pcieport 0000:00:1c.4: eth1: 0x00000470: 0x80000000, 0x00000000, 0x00000000, 0x50000002
[ 1060.661523] pcieport 0000:00:1c.4: eth1: 0x00000480: 0x42000000, 0x7fffffff, 0x06000004, 0x7fffffff
[ 1060.661676] pcieport 0000:00:1c.4: eth1: 0x00000500: 0x00000008, 0x00000002, 0x00000000, 0x00000000
[ 1060.661840] pcieport 0000:00:1c.4: eth1: 0x00000590: 0x00e00000, 0x00000000, 0x00000000, 0x00000000
[ 1060.661997] pcieport 0000:00:1c.4: eth1: 0x000005b0: 0x00000000, 0x00000008, 0x00000000, 0x00000000
[ 1060.662156] pcieport 0000:00:1c.4: eth1: 0x000005c0: 0x03a0a0b0, 0x39800fef, 0x00000000, 0x00000000
[ 1060.662309] pcieport 0000:00:1c.4: eth1: 0x00000600: 0xffffffff, 0x00f80011, 0x00000000, 0x00001f04
[ 1060.662461] pcieport 0000:00:1c.4: eth1: 0x00000610: 0xffffffff, 0x00000000, 0x07c00004, 0x00000000
[ 1060.662619] pcieport 0000:00:1c.4: eth1: 0x00000620: 0x00000040, 0x00000000, 0x00000000, 0x00000000
[ 1060.662776] pcieport 0000:00:1c.4: eth1: 0x00000630: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[ 1060.662934] pcieport 0000:00:1c.4: eth1: 0x00000640: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[ 1060.663087] pcieport 0000:00:1c.4: eth1: 0x00000650: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[ 1060.663245] pcieport 0000:00:1c.4: eth1: 0x00000660: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[ 1060.663403] pcieport 0000:00:1c.4: eth1: 0x00000670: 0x5f865437, 0xe4ac62cc, 0x50103a45, 0x36621985
[ 1060.663560] pcieport 0000:00:1c.4: eth1: 0x00000680: 0xbf14c0e8, 0x1bc27a1e, 0x84f4b556, 0x094ea6fe
[ 1060.663714] pcieport 0000:00:1c.4: eth1: 0x00000690: 0x7dda01e7, 0xc04d7481, 0x00000000, 0x00000000
[ 1060.663871] pcieport 0000:00:1c.4: eth1: 0x000006c0: 0x00000000, 0x00000000, 0x04000000, 0x00000000
[ 1060.664030] pcieport 0000:00:1c.4: eth1: 0x00000800: 0x00000000, 0xffffffff, 0x00000000, 0x00000000
[ 1060.664195] pcieport 0000:00:1c.4: eth1: 0x00000810: 0x00000000, 0xffffffff, 0x00000000, 0x00000000
[ 1060.664350] pcieport 0000:00:1c.4: eth1: 0x00000820: 0x00000000, 0x00000000, 0xffffffff, 0x00000000
[ 1060.664508] pcieport 0000:00:1c.4: eth1: 0x00000830: 0x00000000, 0xffffffff, 0xffffffff, 0xffffffff
[ 1060.664666] pcieport 0000:00:1c.4: eth1: 0x00000840: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
[ 1060.664824] pcieport 0000:00:1c.4: eth1: 0x00000850: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
[ 1060.664981] pcieport 0000:00:1c.4: eth1: 0x00000860: 0xffffffff, 0xffffffff, 0xffffffff, 0x00000000
[ 1060.665140] pcieport 0000:00:1c.4: eth1: 0x000008f0: 0x00000001, 0x00000000, 0x00000000, 0x00000000
[ 1060.665297] pcieport 0000:00:1c.4: eth1: 0x00000900: 0x17ae8430, 0xffffffff, 0x00000000, 0x00000000
[ 1060.665452] pcieport 0000:00:1c.4: eth1: 0x00000910: 0x00000000, 0xffffffff, 0x00000000, 0x00000000
[ 1060.665610] pcieport 0000:00:1c.4: eth1: 0x00000920: 0x00000000, 0x00000000, 0xffffffff, 0x00000000
[ 1060.665777] pcieport 0000:00:1c.4: eth1: 0x00000930: 0x00000000, 0xffffffff, 0xffffffff, 0xffffffff
[ 1060.665935] pcieport 0000:00:1c.4: eth1: 0x00000940: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
[ 1060.666094] pcieport 0000:00:1c.4: eth1: 0x00000950: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
[ 1060.666252] pcieport 0000:00:1c.4: eth1: 0x00000960: 0xffffffff, 0xffffffff, 0xffffffff, 0x00047528
[ 1060.666411] pcieport 0000:00:1c.4: eth1: 0x00000980: 0x0000006a, 0x00000000, 0x00000000, 0x0000caf7
[ 1060.666564] pcieport 0000:00:1c.4: eth1: 0x00000990: 0x00000002, 0x00000000, 0x00000000, 0x00000000
[ 1060.666722] pcieport 0000:00:1c.4: eth1: 0x00000c00: 0x0000000a, 0x00000000, 0x00000003, 0x00000001
[ 1060.666881] pcieport 0000:00:1c.4: eth1: 0x00000c10: 0x00000000, 0x00000007, 0x00000000, 0x00360000
[ 1060.667039] pcieport 0000:00:1c.4: eth1: 0x00000c80: 0x00301416, 0x00000000, 0x00000000, 0x00000000
[ 1060.667197] pcieport 0000:00:1c.4: eth1: 0x00000ce0: 0xd97570fa, 0x0000001f, 0x00000036, 0x00041008
[ 1060.667355] pcieport 0000:00:1c.4: eth1: 0x00000cf0: 0x00000000, 0x5000005e, 0x00000000, 0x00000000
[ 1060.667513] pcieport 0000:00:1c.4: eth1: 0x00001000: 0x00000002, 0x00000000, 0xa0004955, 0x00000000
[ 1060.667671] pcieport 0000:00:1c.4: eth1: 0x00001010: 0x01d81d81, 0x00004955, 0x00000000, 0x00000000
[ 1060.667828] pcieport 0000:00:1c.4: eth1: 0x00001400: 0x00000006, 0x00000000, 0x00000000, 0x00000000
[ 1060.667987] pcieport 0000:00:1c.4: eth1: 0x00001440: 0x000001df, 0x000001df, 0x000001df, 0x000001df
[ 1060.668145] pcieport 0000:00:1c.4: eth1: 0x00001450: 0x000001df, 0x000001df, 0x000001df, 0x000001df
[ 1060.668302] pcieport 0000:00:1c.4: eth1: 0x00001460: 0x000001df, 0x000001df, 0x000001df, 0x000001df
[ 1060.668460] pcieport 0000:00:1c.4: eth1: 0x00001470: 0x000001df, 0x000001df, 0x000001df, 0x000001df
[ 1060.668617] pcieport 0000:00:1c.4: eth1: 0x00001480: 0x00001114, 0x00000000, 0x00000000, 0x00000000
[ 1060.668775] pcieport 0000:00:1c.4: eth1: 0x00001800: 0x00000016, 0x00000000, 0x0000000d, 0x00000000
[ 1060.668933] pcieport 0000:00:1c.4: eth1: 0x00001830: 0x00000000, 0x00000000, 0x00000000, 0x3fb48000
[ 1060.669091] pcieport 0000:00:1c.4: eth1: 0x00001840: 0x3fb48000, 0x00000020, 0x10000211, 0xc0000001
[ 1060.669249] pcieport 0000:00:1c.4: eth1: 0x00001850: 0x0000001f, 0x00000000, 0x00004000, 0x000d000d
[ 1060.669407] pcieport 0000:00:1c.4: eth1: 0x00001860: 0x0d000300, 0x00000000, 0x3fb48000, 0x00000020
[ 1060.669564] pcieport 0000:00:1c.4: eth1: 0x00001c00: 0x00000002, 0x00000000, 0x00000000, 0x00000000
[ 1060.669722] pcieport 0000:00:1c.4: eth1: 0x00002000: 0x00000002, 0x00000000, 0x00000000, 0x00000000
[ 1060.669884] pcieport 0000:00:1c.4: eth1: 0x00002010: 0x00000181, 0x00000001, 0x00780003, 0x00000000
[ 1060.670044] pcieport 0000:00:1c.4: eth1: 0x00002100: 0x000cca66, 0x000cca66, 0x00000000, 0x00000000
[ 1060.670202] pcieport 0000:00:1c.4: eth1: 0x00002110: 0x000cca66, 0x000cca66, 0x00000000, 0x00000000
[ 1060.670423] pcieport 0000:00:1c.4: eth1: 0x00002120: 0x000cca66, 0x000cca66, 0x00000000, 0x00000000
[ 1060.670581] pcieport 0000:00:1c.4: eth1: 0x00002130: 0x000cca66, 0x000cca66, 0x00000000, 0x00000000
[ 1060.670736] pcieport 0000:00:1c.4: eth1: 0x00002140: 0x000cca66, 0x000cca66, 0x00000000, 0x00000000
[ 1060.670893] pcieport 0000:00:1c.4: eth1: 0x00002150: 0x000cca66, 0x000cca66, 0x00000000, 0x00000000
[ 1060.671050] pcieport 0000:00:1c.4: eth1: 0x00002160: 0x000cca66, 0x000cca66, 0x00000000, 0x00000000
[ 1060.671207] pcieport 0000:00:1c.4: eth1: 0x00002170: 0x000cca66, 0x000cca66, 0x00000000, 0x00000000
[ 1060.682692] pcieport 0000:00:1c.4: eth1: 0x00002180: 0x000cca66, 0x000cca66, 0x00000000, 0x00000000
[ 1060.682849] pcieport 0000:00:1c.4: eth1: 0x00002190: 0x000cca66, 0x000cca66, 0x00000000, 0x00000000
[ 1060.683009] pcieport 0000:00:1c.4: eth1: 0x000021a0: 0x000cca66, 0x000cca66, 0x00000000, 0x00000000
[ 1060.683162] pcieport 0000:00:1c.4: eth1: 0x000021b0: 0x000cca66, 0x000cca66, 0x00000000, 0x00000000
[ 1060.683321] pcieport 0000:00:1c.4: eth1: 0x000021c0: 0x000cca66, 0x000cca66, 0x00000000, 0x00000000
[ 1060.683480] pcieport 0000:00:1c.4: eth1: 0x000021d0: 0x000cca66, 0x000cca66, 0x00000000, 0x00000000
[ 1060.683638] pcieport 0000:00:1c.4: eth1: 0x000021e0: 0x000cca66, 0x000cca66, 0x00000000, 0x00000000
[ 1060.683791] pcieport 0000:00:1c.4: eth1: 0x000021f0: 0x000cca66, 0x000cca66, 0x00000000, 0x00000000
[ 1060.683945] pcieport 0000:00:1c.4: eth1: 0x00002200: 0x000f37ab, 0x00000000, 0x00000000, 0x00000000
[ 1060.684103] pcieport 0000:00:1c.4: eth1: 0x00002250: 0x00000045, 0x00000000, 0x00000000, 0x00000000
[ 1060.684269] pcieport 0000:00:1c.4: eth1: 0x00002400: 0x00010012, 0x00000000, 0x00200001, 0x00000000
[ 1060.684423] pcieport 0000:00:1c.4: eth1: 0x00002410: 0x0000000f, 0x00005d00, 0x00000000, 0x00000000
[ 1060.684658] pcieport 0000:00:1c.4: eth1: 0x00002440: 0x00000000, 0x00000000, 0x00000002, 0x00044400
[ 1060.684817] pcieport 0000:00:1c.4: eth1: 0x00002450: 0x00000020, 0x3f8d0000, 0x08001800, 0x00040000
[ 1060.684975] pcieport 0000:00:1c.4: eth1: 0x00002470: 0x00000000, 0x00000766, 0x00000000, 0x00000000
[ 1060.685133] pcieport 0000:00:1c.4: eth1: 0x00002500: 0x00000000, 0x00000000, 0x00000002, 0x00044800
[ 1060.685291] pcieport 0000:00:1c.4: eth1: 0x00002510: 0x00000000, 0x00000000, 0x00000002, 0x00040400
[ 1060.685451] pcieport 0000:00:1c.4: eth1: 0x00002520: 0x00000000, 0x00000000, 0x00000002, 0x00044c00
[ 1060.685608] pcieport 0000:00:1c.4: eth1: 0x00002530: 0x00000000, 0x00000000, 0x00000002, 0x00040800
[ 1060.685780] pcieport 0000:00:1c.4: eth1: 0x00002540: 0x00000000, 0x00000000, 0x00000002, 0x00045000
[ 1060.685939] pcieport 0000:00:1c.4: eth1: 0x00002550: 0x00000000, 0x00000000, 0x00000002, 0x00040c00
[ 1060.686098] pcieport 0000:00:1c.4: eth1: 0x00002560: 0x00000000, 0x00000000, 0x00000002, 0x00045400
[ 1060.686256] pcieport 0000:00:1c.4: eth1: 0x00002570: 0x00000000, 0x00000000, 0x00000002, 0x00041000
[ 1060.686414] pcieport 0000:00:1c.4: eth1: 0x00002580: 0x00000000, 0x00000000, 0x00000002, 0x00045800
[ 1060.686572] pcieport 0000:00:1c.4: eth1: 0x00002590: 0x00000000, 0x00000000, 0x00000002, 0x00041400
[ 1060.686730] pcieport 0000:00:1c.4: eth1: 0x000025a0: 0x00000000, 0x00000000, 0x00000002, 0x00045c00
[ 1060.686888] pcieport 0000:00:1c.4: eth1: 0x000025b0: 0x00000000, 0x00000000, 0x00000002, 0x00041800
[ 1060.687046] pcieport 0000:00:1c.4: eth1: 0x000025c0: 0x00000000, 0x00000000, 0x00000002, 0x00046000
[ 1060.687204] pcieport 0000:00:1c.4: eth1: 0x000025d0: 0x00000000, 0x00000000, 0x00000002, 0x00041c00
[ 1060.687358] pcieport 0000:00:1c.4: eth1: 0x000025e0: 0x00000000, 0x00000000, 0x00000002, 0x00046400
[ 1060.687510] pcieport 0000:00:1c.4: eth1: 0x000025f0: 0x00000000, 0x00000000, 0x00000002, 0x00042000
[ 1060.687665] pcieport 0000:00:1c.4: eth1: 0x00002600: 0x00000000, 0x00000000, 0x00000002, 0x00046800
[ 1060.687823] pcieport 0000:00:1c.4: eth1: 0x00002610: 0x00000000, 0x00000000, 0x00000002, 0x00042400
[ 1060.687975] pcieport 0000:00:1c.4: eth1: 0x00002620: 0x00000000, 0x00000000, 0x00000002, 0x00046c00
[ 1060.688125] pcieport 0000:00:1c.4: eth1: 0x00002630: 0x00000000, 0x00000000, 0x00000002, 0x00042800
[ 1060.688283] pcieport 0000:00:1c.4: eth1: 0x00002640: 0x00000000, 0x00000000, 0x00000002, 0x00047000
[ 1060.688441] pcieport 0000:00:1c.4: eth1: 0x00002650: 0x00000000, 0x00000000, 0x00000002, 0x00042c00
[ 1060.688598] pcieport 0000:00:1c.4: eth1: 0x00002660: 0x00000000, 0x00000000, 0x00000002, 0x00047400
[ 1060.688751] pcieport 0000:00:1c.4: eth1: 0x00002670: 0x00000000, 0x00000000, 0x00000002, 0x00043000
[ 1060.688908] pcieport 0000:00:1c.4: eth1: 0x00002680: 0x00000000, 0x00000000, 0x00000002, 0x00047800
[ 1060.689066] pcieport 0000:00:1c.4: eth1: 0x00002690: 0x00000000, 0x00000000, 0x00000002, 0x00043400
[ 1060.689224] pcieport 0000:00:1c.4: eth1: 0x000026a0: 0x00000000, 0x00000000, 0x00000002, 0x00047c00
[ 1060.689374] pcieport 0000:00:1c.4: eth1: 0x000026b0: 0x00000000, 0x00000000, 0x00000002, 0x00043800
[ 1060.689531] pcieport 0000:00:1c.4: eth1: 0x000026c0: 0x00000000, 0x00000000, 0x00000002, 0x00048000
[ 1060.689689] pcieport 0000:00:1c.4: eth1: 0x000026d0: 0x00000000, 0x00000000, 0x00000002, 0x00043c00
[ 1060.689855] pcieport 0000:00:1c.4: eth1: 0x000026e0: 0x00000000, 0x00000000, 0x00000002, 0x00048400
[ 1060.690012] pcieport 0000:00:1c.4: eth1: 0x000026f0: 0x00000000, 0x00000000, 0x00000002, 0x00044000
[ 1060.690170] pcieport 0000:00:1c.4: eth1: 0x00002800: 0x00000006, 0x00000000, 0x00000000, 0x00000000
[ 1060.690329] pcieport 0000:00:1c.4: eth1: 0x00002c00: 0x00000006, 0x00000000, 0x00000000, 0x000007e6
[ 1060.690482] pcieport 0000:00:1c.4: eth1: 0x00002c10: 0x00000000, 0x00000000, 0x00000019, 0x0000000c
[ 1060.690639] pcieport 0000:00:1c.4: eth1: 0x00002c20: 0x00000001, 0x00000000, 0x00000000, 0x00000000
[ 1060.690797] pcieport 0000:00:1c.4: eth1: 0x00002d00: 0x00000080, 0x00000040, 0x00000000, 0x00000000
[ 1060.690952] pcieport 0000:00:1c.4: eth1: 0x00003000: 0x00000006, 0x00000000, 0x00000000, 0x000007e6
[ 1060.691108] pcieport 0000:00:1c.4: eth1: 0x00003400: 0x00000004, 0x00000000, 0x00000000, 0x00000000
[ 1060.691267] pcieport 0000:00:1c.4: eth1: 0x00003600: 0x00004600, 0x00170000, 0x00110000, 0x00000000
[ 1060.691420] pcieport 0000:00:1c.4: eth1: 0x00003610: 0x00170000, 0x00000000, 0x00130000, 0x00000000
[ 1060.691578] pcieport 0000:00:1c.4: eth1: 0x00003620: 0x00110011, 0x00000000, 0x00000000, 0x40032080
[ 1060.691735] pcieport 0000:00:1c.4: eth1: 0x00003630: 0x00800000, 0x87748774, 0x02c01000, 0x00000000
[ 1060.691893] pcieport 0000:00:1c.4: eth1: 0x00003640: 0x00000000, 0x00000000, 0x00000020, 0x00000019
[ 1060.692051] pcieport 0000:00:1c.4: eth1: 0x00003650: 0x00000171, 0x000f03ff, 0x05720000, 0x00000000
[ 1060.692207] pcieport 0000:00:1c.4: eth1: 0x00003660: 0x00000000, 0x00000000, 0x02000000, 0x00000202
[ 1060.692365] pcieport 0000:00:1c.4: eth1: 0x00003670: 0x00000000, 0xfef7bff3, 0x00000000, 0x00000020
[ 1060.692523] pcieport 0000:00:1c.4: eth1: 0x00003680: 0x30018010, 0x00000000, 0x00000000, 0x00000000
[ 1060.692680] pcieport 0000:00:1c.4: eth1: 0x000036a0: 0x000001a0, 0x00000000, 0x00000000, 0x00000000
[ 1060.692837] pcieport 0000:00:1c.4: eth1: 0x000036b0: 0x00000000, 0x07ff07ff, 0x07ff07ff, 0x01000004
[ 1060.692995] pcieport 0000:00:1c.4: eth1: 0x000036c0: 0xfffffffe, 0x00000000, 0x00000000, 0x003d16ec
[ 1060.693152] pcieport 0000:00:1c.4: eth1: 0x000036d0: 0x0000019d, 0x00000000, 0x00000000, 0x00004916
[ 1060.693310] pcieport 0000:00:1c.4: eth1: 0x000036f0: 0x00000000, 0x00000000, 0x00000000, 0x00011b01
[ 1060.693468] pcieport 0000:00:1c.4: eth1: 0x00003800: 0x00000001, 0x00000000, 0x0000000e, 0x0516028b
[ 1060.693626] pcieport 0000:00:1c.4: eth1: 0x00003810: 0x0000016f, 0x00000093, 0x00000000, 0x00000000
[ 1060.693793] pcieport 0000:00:1c.4: eth1: 0x00003c00: 0x00000306, 0x00000000, 0x00000000, 0x00000048
[ 1060.693949] pcieport 0000:00:1c.4: eth1: 0x00003c10: 0x00000000, 0x00000035, 0x00000000, 0x00000000
[ 1060.694107] pcieport 0000:00:1c.4: eth1: 0x00003c20: 0x00000000, 0x00000005, 0x00000000, 0x00000000
[ 1060.694266] pcieport 0000:00:1c.4: eth1: 0x00003c30: 0x00000000, 0x00000000, 0x00000010, 0x1053b000
[ 1060.694426] pcieport 0000:00:1c.4: eth1: 0x00003c40: 0x00000000, 0x00000b00, 0x00000000, 0x00000000
[ 1060.694584] pcieport 0000:00:1c.4: eth1: 0x00003c50: 0x00000000, 0x00000766, 0x00000000, 0x00000000
[ 1060.694743] pcieport 0000:00:1c.4: eth1: 0x00003c80: 0x00000024, 0x0000073a, 0x00000000, 0x00000008
[ 1060.694900] pcieport 0000:00:1c.4: eth1: 0x00003cc0: 0x000001d8, 0x00000000, 0x00000000, 0x00000000
[ 1060.695057] pcieport 0000:00:1c.4: eth1: 0x00003cd0: 0x00000000, 0x0000000f, 0x00000000, 0x00000000
[ 1060.695215] pcieport 0000:00:1c.4: eth1: 0x00003d00: 0x00000010, 0x2ff6b000, 0x00000010, 0x2fde2000
[ 1060.695373] pcieport 0000:00:1c.4: eth1: 0x00003d10: 0x00000010, 0x10077000, 0x00000010, 0x104e0000
[ 1060.695526] pcieport 0000:00:1c.4: eth1: 0x00003d80: 0x00000014, 0x00000000, 0x00000005, 0x00000000
[ 1060.695681] pcieport 0000:00:1c.4: eth1: 0x00003d90: 0x00000005, 0x00000000, 0x00000014, 0x00000000
[ 1060.695839] pcieport 0000:00:1c.4: eth1: 0x00003da0: 0x00000005, 0x00000000, 0x00000005, 0x00000000
[ 1060.695997] pcieport 0000:00:1c.4: eth1: 0x00003db0: 0x00000014, 0x00000000, 0x00000005, 0x00000000
[ 1060.696154] pcieport 0000:00:1c.4: eth1: 0x00003dc0: 0x00000005, 0x00000000, 0x00000014, 0x00000000
[ 1060.696307] pcieport 0000:00:1c.4: eth1: 0x00003dd0: 0x00000005, 0x00000000, 0x00000005, 0x00000000
[ 1060.696465] pcieport 0000:00:1c.4: eth1: 0x00003fc0: 0x000d9da2, 0x00000000, 0x00000000, 0x00000000
[ 1060.696623] pcieport 0000:00:1c.4: eth1: 0x00004000: 0x00000002, 0x00000000, 0x0019e4fc, 0x0017b330
[ 1060.696781] pcieport 0000:00:1c.4: eth1: 0x00004010: 0x00000000, 0x00266012, 0x00000440, 0x00859842
[ 1060.708239] pcieport 0000:00:1c.4: eth1: 0x00004020: 0x00000000, 0x00000000, 0x00000010, 0x00000000
[ 1060.708396] pcieport 0000:00:1c.4: eth1: 0x00004030: 0x00000010, 0x00000050, 0x00000000, 0x00000000
[ 1060.708556] pcieport 0000:00:1c.4: eth1: 0x00004040: 0x00000000, 0x00000000, 0x01083a20, 0x00000000
[ 1060.708715] pcieport 0000:00:1c.4: eth1: 0x00004050: 0x00000000, 0x00000000, 0x00266010, 0x00434002
[ 1060.708873] pcieport 0000:00:1c.4: eth1: 0x00004060: 0x00440000, 0x00000000, 0x00000000, 0x00000000
[ 1060.709032] pcieport 0000:00:1c.4: eth1: 0x00004400: 0x00000016, 0x00000000, 0x00010000, 0x0000a000
[ 1060.709189] pcieport 0000:00:1c.4: eth1: 0x00004410: 0x00000000, 0x0000002a, 0x000000a0, 0x00000000
[ 1060.709343] pcieport 0000:00:1c.4: eth1: 0x00004420: 0x0000003d, 0x00000000, 0x00000000, 0x00000000
[ 1060.709501] pcieport 0000:00:1c.4: eth1: 0x00004440: 0x00000000, 0x00000000, 0x00000000, 0x04915458
[ 1060.709658] pcieport 0000:00:1c.4: eth1: 0x00004450: 0x0002073f, 0x00670068, 0x00000000, 0x00000000
[ 1060.709823] pcieport 0000:00:1c.4: eth1: 0x00004800: 0x380303fe, 0x00000000, 0x00000000, 0x00000120
[ 1060.709978] pcieport 0000:00:1c.4: eth1: 0x00004810: 0x00000000, 0x0000000a, 0x00369c58, 0x00002a28
[ 1060.710137] pcieport 0000:00:1c.4: eth1: 0x00004820: 0x00000040, 0x00000000, 0xf0330000, 0x46e54b68
[ 1060.710296] pcieport 0000:00:1c.4: eth1: 0x00004830: 0x00000000, 0x00000356, 0x00000356, 0x00000000
[ 1060.710454] pcieport 0000:00:1c.4: eth1: 0x00004840: 0x000efca8, 0x000efca8, 0x000e2200, 0x40364c27
[ 1060.710609] pcieport 0000:00:1c.4: eth1: 0x00004850: 0xe06ac5ea, 0x83f905ea, 0x34333433, 0x00000000
[ 1060.710767] pcieport 0000:00:1c.4: eth1: 0x00004860: 0x00000040, 0x110002e1, 0x00100800, 0xea040008
[ 1060.710926] pcieport 0000:00:1c.4: eth1: 0x00004870: 0x05ea0085, 0x003e1820, 0x003e1820, 0x00000000
[ 1060.711084] pcieport 0000:00:1c.4: eth1: 0x00004900: 0x28190404, 0x00305407, 0x00000000, 0x00000000
[ 1060.711242] pcieport 0000:00:1c.4: eth1: 0x00004910: 0x000f001c, 0x00000000, 0x00000000, 0x00000000
[ 1060.711400] pcieport 0000:00:1c.4: eth1: 0x00004a00: 0x180303fe, 0x00200000, 0x00200020, 0x353a0000
[ 1060.711557] pcieport 0000:00:1c.4: eth1: 0x00004a10: 0x3f8dfc60, 0x008c0904, 0x00200010, 0x00000000
[ 1060.711715] pcieport 0000:00:1c.4: eth1: 0x00004a20: 0x00000000, 0x00000000, 0xf02c0000, 0x3f8dfcc0
[ 1060.711872] pcieport 0000:00:1c.4: eth1: 0x00004a30: 0x00000000, 0x000001b2, 0x000001b2, 0x00000000
[ 1060.712031] pcieport 0000:00:1c.4: eth1: 0x00004a40: 0x3f8dfc40, 0x3f8dfc60, 0x3f8dfc00, 0x3f8dfc20
[ 1060.712191] pcieport 0000:00:1c.4: eth1: 0x00004a50: 0x00200020, 0x00200020, 0x00200020, 0x00200020
[ 1060.712349] pcieport 0000:00:1c.4: eth1: 0x00004a70: 0x28190404, 0x00305407, 0x000f001c, 0x00000000
[ 1060.712507] pcieport 0000:00:1c.4: eth1: 0x00004b00: 0x180303fe, 0x00360003, 0x30380000, 0x05b401e0
[ 1060.712664] pcieport 0000:00:1c.4: eth1: 0x00004b10: 0x00360042, 0x05b40381, 0x0036dc54, 0x00196e8c
[ 1060.712817] pcieport 0000:00:1c.4: eth1: 0x00004b20: 0x05b40057, 0x022b6004, 0xe76af0f9, 0x0f360038
[ 1060.712967] pcieport 0000:00:1c.4: eth1: 0x00004b30: 0xfffff321, 0x24b405b8, 0x31b1acf9, 0x0f360038
[ 1060.713130] pcieport 0000:00:1c.4: eth1: 0x00004b40: 0xfffff8d9, 0x24b405b8, 0x6c6c564f, 0xf3ec4a4a
[ 1060.713288] pcieport 0000:00:1c.4: eth1: 0x00004b50: 0xf0330000, 0xe76af130, 0xaf040000, 0xf514a2ab
[ 1060.713441] pcieport 0000:00:1c.4: eth1: 0x00004b60: 0x48b105b8, 0xfffff320, 0x410005b4, 0x1dd95010
[ 1060.713591] pcieport 0000:00:1c.4: eth1: 0x00004b70: 0xf0320000, 0x31b1ad30, 0xaf040000, 0x0a0000ff
[ 1060.713758] pcieport 0000:00:1c.4: eth1: 0x00004b80: 0x00000056, 0x110002e1, 0x00100800, 0xea040008
[ 1060.713918] pcieport 0000:00:1c.4: eth1: 0x00004b90: 0x05ea0085, 0x28190404, 0x00305407, 0x000f001c
[ 1060.714076] pcieport 0000:00:1c.4: eth1: 0x00004ba0: 0x0050004d, 0x00000056, 0x110002e1, 0x000f001c
[ 1060.714229] pcieport 0000:00:1c.4: eth1: 0x00004bb0: 0x31b1a8f9, 0x31b1a4f9, 0x5be28f81, 0x5be27201
[ 1060.714388] pcieport 0000:00:1c.4: eth1: 0x00004bc0: 0xe76af0fa, 0xfffff324, 0x31b1acfa, 0xfffff8d8
[ 1060.714550] pcieport 0000:00:1c.4: eth1: 0x00004bd0: 0x31b1a8fa, 0x31b1a4fa, 0x5be28d70, 0x5be27000
[ 1060.714708] pcieport 0000:00:1c.4: eth1: 0x00004be0: 0x05b40036, 0x05b40036, 0x00360036, 0x03240290
[ 1060.714866] pcieport 0000:00:1c.4: eth1: 0x00004bf0: 0x48b105b8, 0xfffff8d8, 0x010005b4, 0x000053b8
[ 1060.715021] pcieport 0000:00:1c.4: eth1: 0x00004c00: 0x200003fe, 0x00000000, 0x00000000, 0x00000000
[ 1060.715179] pcieport 0000:00:1c.4: eth1: 0x00004c10: 0x0000002a, 0x00000000, 0x00000006, 0x00000000
[ 1060.715337] pcieport 0000:00:1c.4: eth1: 0x00004c20: 0x00000000, 0x00000000, 0x00000000, 0x00000006
[ 1060.715488] pcieport 0000:00:1c.4: eth1: 0x00004c30: 0x00000000, 0x00000000, 0x00108000, 0x00000000
[ 1060.715646] pcieport 0000:00:1c.4: eth1: 0x00004c40: 0x00000020, 0x00000000, 0x001d0020, 0x00040020
[ 1060.715804] pcieport 0000:00:1c.4: eth1: 0x00004c50: 0xff739023, 0x00023007, 0x0800073a, 0x21661010
[ 1060.715963] pcieport 0000:00:1c.4: eth1: 0x00004c60: 0x00000020, 0x00000000, 0x00000000, 0x00000000
[ 1060.716119] pcieport 0000:00:1c.4: eth1: 0x00005000: 0x00009800, 0x80008000, 0x00000000, 0x00000000
[ 1060.716276] pcieport 0000:00:1c.4: eth1: 0x00005010: 0x00000000, 0x00000000, 0x00000000, 0x08002084
[ 1060.716434] pcieport 0000:00:1c.4: eth1: 0x00005020: 0x24634521, 0x00000000, 0x00000000, 0x40000020
[ 1060.716592] pcieport 0000:00:1c.4: eth1: 0x00005030: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[ 1060.716749] pcieport 0000:00:1c.4: eth1: 0x00005040: 0x00000000, 0x00000000, 0x08002092, 0x00000000
[ 1060.716901] pcieport 0000:00:1c.4: eth1: 0x00005080: 0x00009800, 0x80004000, 0x00000000, 0x00000000
[ 1060.717059] pcieport 0000:00:1c.4: eth1: 0x00005090: 0x00000000, 0x00000000, 0x00000000, 0x08000078
[ 1060.717217] pcieport 0000:00:1c.4: eth1: 0x000050a0: 0x10a00007, 0x00000000, 0x00000000, 0x40000020
[ 1060.717375] pcieport 0000:00:1c.4: eth1: 0x000050b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[ 1060.717533] pcieport 0000:00:1c.4: eth1: 0x000050c0: 0x00000000, 0x00000000, 0x08001fb4, 0x00000000
[ 1060.717687] pcieport 0000:00:1c.4: eth1: 0x00005100: 0x00009800, 0x80000000, 0x00000000, 0x00000000
[ 1060.717855] pcieport 0000:00:1c.4: eth1: 0x00005110: 0x00000000, 0x00000000, 0x00000000, 0x08002084
[ 1060.718015] pcieport 0000:00:1c.4: eth1: 0x00005120: 0x3c038000, 0x00000000, 0x00000000, 0x40000020
[ 1060.718173] pcieport 0000:00:1c.4: eth1: 0x00005130: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[ 1060.718331] pcieport 0000:00:1c.4: eth1: 0x00005140: 0x00000000, 0x00000000, 0x08001fea, 0x00000000
[ 1060.718489] pcieport 0000:00:1c.4: eth1: 0x00005180: 0x00009800, 0x80004000, 0x00000000, 0x00000000
[ 1060.718647] pcieport 0000:00:1c.4: eth1: 0x00005190: 0x00000000, 0x00000000, 0x00000000, 0x080019e4
[ 1060.718805] pcieport 0000:00:1c.4: eth1: 0x000051a0: 0x30422000, 0x00000000, 0x00000000, 0x40000020
[ 1060.718963] pcieport 0000:00:1c.4: eth1: 0x000051b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[ 1060.719120] pcieport 0000:00:1c.4: eth1: 0x000051c0: 0x00000000, 0x00000000, 0x08002830, 0x00000000
[ 1060.719279] pcieport 0000:00:1c.4: eth1: 0x00005200: 0x00000000, 0x14600027, 0x3c038000, 0x08006f98
[ 1060.719436] pcieport 0000:00:1c.4: eth1: 0x00005210: 0x14600027, 0x00000015, 0x144001e5, 0x00000000
[ 1060.719594] pcieport 0000:00:1c.4: eth1: 0x00005220: 0x01020000, 0x144001e5, 0x00000000, 0x00000b50
[ 1060.719753] pcieport 0000:00:1c.4: eth1: 0x00005230: 0xc0010000, 0x3c038000, 0x08006f80, 0x14600027
[ 1060.719910] pcieport 0000:00:1c.4: eth1: 0x00005240: 0x3c038000, 0x01020000, 0x14600027, 0x3c038000
[ 1060.720068] pcieport 0000:00:1c.4: eth1: 0x00005250: 0x00000000, 0x14600027, 0x3c038000, 0x08006f98
[ 1060.720226] pcieport 0000:00:1c.4: eth1: 0x00005260: 0x14600027, 0x00000000, 0x144001e5, 0x00000000
[ 1060.720383] pcieport 0000:00:1c.4: eth1: 0x00005270: 0xb49a89ab, 0x144001e5, 0x80000000, 0x01020000
[ 1060.720538] pcieport 0000:00:1c.4: eth1: 0x00005280: 0x00009800, 0x80004000, 0x00000000, 0x00000000
[ 1060.720697] pcieport 0000:00:1c.4: eth1: 0x00005290: 0x00000000, 0x00000000, 0x00000000, 0x080019dc
[ 1060.720854] pcieport 0000:00:1c.4: eth1: 0x000052a0: 0x30422000, 0x00000000, 0x00000000, 0x40000020
[ 1060.721008] pcieport 0000:00:1c.4: eth1: 0x000052b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[ 1060.721163] pcieport 0000:00:1c.4: eth1: 0x000052c0: 0x00000000, 0x00000000, 0x08002830, 0x00000000
[ 1060.721321] pcieport 0000:00:1c.4: eth1: 0x00005300: 0x00009800, 0x80004000, 0x00000000, 0x00000000
[ 1060.721479] pcieport 0000:00:1c.4: eth1: 0x00005310: 0x00000000, 0x00000000, 0x00000000, 0x08001fdc
[ 1060.721636] pcieport 0000:00:1c.4: eth1: 0x00005320: 0x8f633630, 0x00000000, 0x00000000, 0x40000020
[ 1060.721797] pcieport 0000:00:1c.4: eth1: 0x00005330: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[ 1060.721956] pcieport 0000:00:1c.4: eth1: 0x00005340: 0x00000000, 0x00000000, 0x08000088, 0x00000000
[ 1060.722114] pcieport 0000:00:1c.4: eth1: 0x00005380: 0x00009800, 0x80000000, 0x00000000, 0x00000000
[ 1060.722272] pcieport 0000:00:1c.4: eth1: 0x00005390: 0x00000000, 0x00000000, 0x00000000, 0x08002830
[ 1060.733762] pcieport 0000:00:1c.4: eth1: 0x000053a0: 0x00000000, 0x00000000, 0x00000000, 0x40000020
[ 1060.733918] pcieport 0000:00:1c.4: eth1: 0x000053b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[ 1060.734076] pcieport 0000:00:1c.4: eth1: 0x000053c0: 0x00000000, 0x00000000, 0x08002dc8, 0x00000000
[ 1060.734230] pcieport 0000:00:1c.4: eth1: 0x00005800: 0x10000000, 0x10000000, 0x25000000, 0x00000000
[ 1060.734389] pcieport 0000:00:1c.4: eth1: 0x00005810: 0x64000000, 0x00000000, 0x01000000, 0x00000000
[ 1060.734554] pcieport 0000:00:1c.4: eth1: 0x00005820: 0x08000000, 0x00000000, 0x00000000, 0x00000000
[ 1060.734712] pcieport 0000:00:1c.4: eth1: 0x00005860: 0x00000000, 0x00000000, 0x0000002e, 0x0000002e
[ 1060.734866] pcieport 0000:00:1c.4: eth1: 0x00005880: 0x00000024, 0x00000024, 0x0000073a, 0x0000073a
[ 1060.735020] pcieport 0000:00:1c.4: eth1: 0x00005890: 0x00000000, 0x00000000, 0x00000008, 0x00000008
[ 1060.735178] pcieport 0000:00:1c.4: eth1: 0x00005900: 0x0000004b, 0x0000004b, 0x00000000, 0x00000000
[ 1060.735337] pcieport 0000:00:1c.4: eth1: 0x00005980: 0x0000000d, 0x00000000, 0x00000000, 0x00000000
[ 1060.735490] pcieport 0000:00:1c.4: eth1: 0x00005a00: 0x000f601f, 0x00000000, 0x00010000, 0x00000000
[ 1060.735648] pcieport 0000:00:1c.4: eth1: 0x00006000: 0x00010082, 0x00000000, 0x00000000, 0x00000000
[ 1060.735806] pcieport 0000:00:1c.4: eth1: 0x00006400: 0x00000000, 0x00000000, 0x00010891, 0xc0000000
[ 1060.735964] pcieport 0000:00:1c.4: eth1: 0x00006410: 0x0a000064, 0x0a000064, 0x00000000, 0x00000000
[ 1060.736117] pcieport 0000:00:1c.4: eth1: 0x00006420: 0x00000000, 0x00000000, 0x00000000, 0x818c0000
[ 1060.736275] pcieport 0000:00:1c.4: eth1: 0x00006430: 0x78000000, 0x14e4165f, 0x04f81028, 0x00020000
[ 1060.736432] pcieport 0000:00:1c.4: eth1: 0x00006440: 0x0000304f, 0x000002e4, 0x00000000, 0x00000000
[ 1060.736590] pcieport 0000:00:1c.4: eth1: 0x000064c0: 0x00000010, 0x00000004, 0x00001004, 0x00000000
[ 1060.736748] pcieport 0000:00:1c.4: eth1: 0x000064d0: 0x00000000, 0x10008d81, 0x00000000, 0x00315e22
[ 1060.736906] pcieport 0000:00:1c.4: eth1: 0x000064e0: 0x00000031, 0x0000001f, 0x00000000, 0x00000000
[ 1060.737063] pcieport 0000:00:1c.4: eth1: 0x000064f0: 0x00000002, 0x00000031, 0x00000000, 0x00000000
[ 1060.737221] pcieport 0000:00:1c.4: eth1: 0x00006500: 0x01e10003, 0xafe933db, 0x0000f01f, 0x00000003
[ 1060.737379] pcieport 0000:00:1c.4: eth1: 0x00006510: 0x0007811b, 0x00058116, 0x00046113, 0x00000000
[ 1060.737537] pcieport 0000:00:1c.4: eth1: 0x00006550: 0x00000001, 0x02800000, 0x00000000, 0x00000000
[ 1060.737695] pcieport 0000:00:1c.4: eth1: 0x000065f0: 0x00000000, 0x00000109, 0x00000000, 0x00000000
[ 1060.737862] pcieport 0000:00:1c.4: eth1: 0x00006800: 0x141b0034, 0x20099082, 0x01069208, 0x3af00bc8
[ 1060.738023] pcieport 0000:00:1c.4: eth1: 0x00006810: 0x01020000, 0xffffffff, 0x00000000, 0x00000000
[ 1060.738181] pcieport 0000:00:1c.4: eth1: 0x00006830: 0xffffffff, 0xffffffff, 0x00000000, 0x00000000
[ 1060.738336] pcieport 0000:00:1c.4: eth1: 0x00006840: 0x00000000, 0x00000001, 0x00000000, 0x00000000
[ 1060.738487] pcieport 0000:00:1c.4: eth1: 0x00006890: 0x00000000, 0x88003800, 0x00000000, 0x00102040
[ 1060.738643] pcieport 0000:00:1c.4: eth1: 0x000068a0: 0x00000020, 0x00000001, 0x03ff03ff, 0x00000000
[ 1060.738801] pcieport 0000:00:1c.4: eth1: 0x000068b0: 0xe0011514, 0x00000000, 0x00000000, 0x00000000
[ 1060.738959] pcieport 0000:00:1c.4: eth1: 0x000068e0: 0x00000000, 0x00000000, 0x00000000, 0x00004978
[ 1060.739110] pcieport 0000:00:1c.4: eth1: 0x000068f0: 0x00ff000e, 0x00ff0000, 0x00000000, 0x04444444
[ 1060.739267] pcieport 0000:00:1c.4: eth1: 0x00006900: 0xa69e9fd8, 0x1378c421, 0x00000000, 0x00000000
[ 1060.739425] pcieport 0000:00:1c.4: eth1: 0x00006920: 0x00000000, 0x00000000, 0x00000001, 0x00000000
[ 1060.739584] pcieport 0000:00:1c.4: eth1: 0x00007000: 0x00000188, 0x00000000, 0x00000000, 0x0004aee0
[ 1060.739742] pcieport 0000:00:1c.4: eth1: 0x00007010: 0x00000078, 0x01c08073, 0x00d70081, 0x03008200
[ 1060.739900] pcieport 0000:00:1c.4: eth1: 0x00007020: 0x00000000, 0x00000000, 0x00000406, 0x10004000
[ 1060.740058] pcieport 0000:00:1c.4: eth1: 0x00007030: 0x000e0000, 0x0004aee4, 0x00170030, 0x00000000
[ 1060.740221] pcieport 0000:00:1c.4: eth1: 0: Host status block [00000001:00000010:(0000:0756:0000):(0000:01d8)]
[ 1060.740383] pcieport 0000:00:1c.4: eth1: 0: NAPI info [00000010:00000010:(004b:01d8:01ff):0000:(002e:0000:0000:0000)]
[ 1060.740543] pcieport 0000:00:1c.4: eth1: 1: Host status block [00000001:00000025:(0000:0000:0000):(0024:0000)]
[ 1060.740705] pcieport 0000:00:1c.4: eth1: 1: NAPI info [00000025:00000025:(0000:0000:01ff):0024:(0024:0024:0000:0000)]
[ 1060.740869] pcieport 0000:00:1c.4: eth1: 2: Host status block [00000001:00000064:(073a:0000:0000):(0000:0000)]
[ 1060.741029] pcieport 0000:00:1c.4: eth1: 2: NAPI info [00000064:00000064:(0000:0000:01ff):073a:(073a:073a:0000:0000)]
[ 1060.741192] pcieport 0000:00:1c.4: eth1: 3: Host status block [00000001:00000001:(0000:0000:0000):(0000:0000)]
[ 1060.741354] pcieport 0000:00:1c.4: eth1: 3: NAPI info [00000001:00000001:(0000:0000:01ff):0000:(0000:0000:0000:0000)]
[ 1060.741612] pcieport 0000:00:1c.4: eth1: 4: Host status block [00000001:00000008:(0000:0000:0008):(0000:0000)]
[ 1060.741786] pcieport 0000:00:1c.4: eth1: 4: NAPI info [00000008:00000008:(0000:0000:01ff):0008:(0008:0008:0000:0000)]
[ 1060.843702] tg3 0000:02:00.1: tg3_stop_block timed out, ofs=1400 enable_bit=2
[ 1060.944947] tg3 0000:02:00.1: tg3_stop_block timed out, ofs=c00 enable_bit=2
[ 1061.019159] pcieport 0000:00:1c.4: eth1: Link is down
[ 1064.536517] pcieport 0000:00:1c.4: eth1: Link is up at 1000 Mbps, full duplex
[ 1064.536536] pcieport 0000:00:1c.4: eth1: Flow control is off for TX and off for RX
[ 1064.536539] pcieport 0000:00:1c.4: eth1: EEE is disabled
[ 1071.188108]  connection1:0: detected conn error (1019)

This sometimes is followed by one or two extra entries like this, and
finally, the system will lock up /reboot.

While debugging, we noticed that the bug goes away when we disable tso,
sg and tx with ethtool. Then, we can run the test over and over for more
than 12 hours...

If you need any additional info, feel free to ask.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1152580

Title:
  [Dell PowerEdge M520] BCM95720 Network disconnects and resets

Status in The Dell PowerEdge project:
  Confirmed
Status in “linux” package in Ubuntu:
  Incomplete

Bug description:
  The server is constantly going on and offline here are messages and hardware Details:
  [    4.394703] tg3 0000:02:00.0: eth0: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 90:b1:1c:0d:d4:aa
  [    4.394713] tg3 0000:02:00.0: eth0: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
  [    4.394720] tg3 0000:02:00.0: eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
  [    4.394726] tg3 0000:02:00.0: eth0: dma_rwctrl[00000001] dma_mask[64-bit]

  The dump to dmesg:
  https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1152580/+attachment/3939814/+files/log

  ProblemType: Bug
  DistroRelease: Ubuntu 12.04
  Package: linux-image-3.2.0-38-generic 3.2.0-38.61
  ProcVersionSignature: Ubuntu 3.2.0-38.61-generic 3.2.37
  Uname: Linux 3.2.0-38-generic x86_64
  AlsaDevices:
   total 0
   crw-rw---T 1 root audio 116,  1 Mar  8 13:26 seq
   crw-rw---T 1 root audio 116, 33 Mar  8 13:26 timer
  AplayDevices: Error: [Errno 2] No such file or directory
  ApportVersion: 2.0.1-0ubuntu17.1
  Architecture: amd64
  ArecordDevices: Error: [Errno 2] No such file or directory
  AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
  CRDA: Error: [Errno 2] No such file or directory
  Date: Fri Mar  8 14:21:45 2013
  HibernationDevice: RESUME=UUID=f9a32355-70b4-462b-a01e-2665b81b2ec0
  InstallationMedia: Ubuntu-Server 12.04.1 LTS "Precise Pangolin" - Release amd64 (20120817.3)
  MachineType: Dell Inc. PowerEdge R520
  MarkForUpload: True
  PciMultimedia:

  ProcFB: 0 VESA VGA
  ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.2.0-38-generic root=UUID=0c0140bd-05eb-4e5c-9596-1cf4dee88ae4 ro
  RelatedPackageVersions:
   linux-restricted-modules-3.2.0-38-generic N/A
   linux-backports-modules-3.2.0-38-generic  N/A
   linux-firmware                            1.79.1
  RfKill: Error: [Errno 2] No such file or directory
  SourcePackage: linux
  StagingDrivers: mei
  UpgradeStatus: No upgrade log present (probably fresh install)
  dmi.bios.date: 08/21/2012
  dmi.bios.vendor: Dell Inc.
  dmi.bios.version: 1.3.5
  dmi.board.name: 051XDX
  dmi.board.vendor: Dell Inc.
  dmi.board.version: A00
  dmi.chassis.type: 23
  dmi.chassis.vendor: Dell Inc.
  dmi.modalias: dmi:bvnDellInc.:bvr1.3.5:bd08/21/2012:svnDellInc.:pnPowerEdgeR520:pvr:rvnDellInc.:rn051XDX:rvrA00:cvnDellInc.:ct23:cvr:
  dmi.product.name: PowerEdge R520
  dmi.sys.vendor: Dell Inc.

To manage notifications about this bug go to:
https://bugs.launchpad.net/dell-poweredge/+bug/1152580/+subscriptions