group.of.nepali.translators team mailing list archive
-
group.of.nepali.translators team
-
Mailing list archive
-
Message #10253
[Bug 1646857] Re: Ubuntu16.10 - EEH on BELL3 adapter fails to recover (serial/tty)
** Changed in: linux (Ubuntu Xenial)
Status: Fix Committed => Fix Released
--
You received this bug notification because you are a member of नेपाली
भाषा समायोजकहरुको समूह, which is subscribed to Xenial.
Matching subscriptions: Ubuntu 16.04 Bugs
https://bugs.launchpad.net/bugs/1646857
Title:
Ubuntu16.10 - EEH on BELL3 adapter fails to recover (serial/tty)
Status in linux package in Ubuntu:
Fix Released
Status in linux source package in Trusty:
Fix Released
Status in linux source package in Vivid:
Fix Released
Status in linux source package in Xenial:
Fix Released
Status in linux source package in Yakkety:
Fix Released
Status in linux source package in Zesty:
Fix Released
Bug description:
---Problem Description---
Error injection on BELL3 serial adapter fails to recover adapter & throws kernel warning/traces. I had to reboot system to reclaim adapter.
---uname output---
Linux ltcalpine-lp4 3.13.0-69-generic #112-Ubuntu SMP Tue Nov 10 16:29:16 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux
Machine Type = IBM,8408-E8E
---Steps to Reproduce---
1. Install librtas & ibm-lop utils
2. Enable EEH in FSP
3. Run workload on serial ports & trigger EEH
i.e
root@ltcalpine-lp4:/home/ubuntu# lspci -v
0000:01:00.0 Serial controller: Digi International Device 00f6 (prog-if 06 [16950])
Subsystem: IBM Device 0422
Flags: fast devsel, IRQ 504
Memory at 3fc200400000 (32-bit, non-prefetchable) [size=16K]
Memory at 3fc200200000 (32-bit, non-prefetchable) [size=2M]
Memory at 3fc200000000 (32-bit, non-prefetchable) [size=2M]
Capabilities: [40] Power Management version 3
Capabilities: [70] Express Endpoint, MSI 00
Capabilities: [b0] MSI-X: Enable- Count=16 Masked-
Capabilities: [100] Device Serial Number 00-30-e0-11-11-00-01-50
Capabilities: [110] Power Budgeting <?>
Kernel driver in use: serial
root@ltcalpine-lp4:/home/ubuntu# ls -l
total 576376
-rw-rw-r-- 1 ubuntu ubuntu 87264 Nov 13 03:51 devices.chrp.base.servicerm_2.5.0.1-15111_ppc64el.deb
-rw-r--r-- 1 ubuntu ubuntu 18726 Nov 13 03:51 dmesg_after_DLPAR.txt
-rw-r--r-- 1 root root 24144 Nov 13 04:20 dmesg-dlpar
-rw-r--r-- 1 root root 32697 Nov 17 07:33 dmesg-eeh
-rw-rw-r-- 1 ubuntu ubuntu 36380 Nov 13 03:51 dynamicrm_2.0.1-3_ppc64el.deb
drwxr-xr-x 6 root root 4096 Nov 13 05:42 ibm-lop-utils
-rw-r--r-- 1 root root 24692 Aug 13 18:47 ibm-lop-utils-0.1.4-1.ppc64le.rpm
-rw-r--r-- 1 root root 27070 Aug 10 18:30 librtas1_1.3.13-1~errinjct.0_ppc64el.deb
-rw-r--r-- 1 root root 29598 Aug 11 18:48 librtas-dev_1.3.13-1~errinjct.0_ppc64el.deb
-rw-rw-r-- 1 ubuntu ubuntu 8040530 Nov 13 03:51 rsct.core_3.2.0.6-15111_ppc64el.deb
-rw-rw-r-- 1 ubuntu ubuntu 771156 Nov 13 03:51 rsct.core.utils_3.2.0.6-15111_ppc64el.deb
-rw-rw-r-- 1 ubuntu ubuntu 219058 Nov 13 03:51 src_3.2.0.6-15111_ppc64el.deb
-rw-r--r-- 1 root root 580864000 Nov 13 00:48 trusty-server-ppc64el.iso
root@ltcalpine-lp4:/home/ubuntu#
root@ltcalpine-lp4:/home/ubuntu# while true ; do dmesg > /dev/ttyS0 ; done &
[1] 1211
root@ltcalpine-lp4:/home/ubuntu# while true ; do dmesg > /dev/ttyS1 ; done &
[2] 1216
root@ltcalpine-lp4:/home/ubuntu#
root@ltcalpine-lp4:/home/ubuntu/ibm-lop-utils/src/errinjct# ./errinjct eeh -v -f 0 -p U78C7.001.RCH0040-P1-C12
Injecting an ioa-bus-error with the following data:
BUS ADDR: 00000000
ADDR MASK: 00000000
CONFIG ADDR: 10000
PHB UNIT_ID: 800000020000019
FUNCTION: 0
Load to PCI Memory Address Space - inject an Address Parity Error
Call to RTAS errinjct succeeded!
If the correct information was provided and there is
activity on the bus, the hardware should hit the error
However, if incorrect information was provided or there
is no bus activity, you may not get a hit.
root@ltcalpine-lp4:/home/ubuntu/ibm-lop-utils/src/errinjct#
Contact Information = backup: mputtash@xxxxxxxxxx
Stack trace output:
[ 158.042962] EEH: Frozen PE#10000 detected on PHB#0
[ 158.042980] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.13.0-69-generic #112-Ubuntu
[ 158.042983] Call Trace:
[ 158.042990] [c000000c7ffef8e0] [c000000000016b00] .show_stack+0x170/0x290 (unreliable)
[ 158.042996] [c000000c7ffef9d0] [c0000000009787d0] .dump_stack+0x88/0xb4
[ 158.043001] [c000000c7ffefa50] [c0000000000365b0] .eeh_dev_check_failure+0x430/0x480
[ 158.043004] [c000000c7ffefb00] [c000000000036684] .eeh_check_failure+0x84/0xe0
[ 158.043009] [c000000c7ffefb90] [c0000000005b3c14] .mem_serial_in+0x94/0xc0
[ 158.043012] [c000000c7ffefc10] [c0000000005b5dec] .serial8250_default_handle_irq+0x3c/0x90
[ 158.043016] [c000000c7ffefc90] [c0000000005b41cc] .serial8250_interrupt+0x6c/0x160
[ 158.043020] [c000000c7ffefd30] [c0000000001074d4] .handle_irq_event_percpu+0x94/0x310
[ 158.043024] [c000000c7ffefe10] [c0000000001077b0] .handle_irq_event+0x60/0xb0
[ 158.043027] [c000000c7ffefe90] [c00000000010c4d4] .handle_fasteoi_irq+0xd4/0x1f0
[ 158.043030] [c000000c7ffeff10] [c00000000001167c] .__do_irq+0xac/0x1c0
[ 158.043034] [c000000c7ffeff90] [c000000000025938] .call_do_irq+0x14/0x24
[ 158.043037] [c00000000164f6d0] [c000000000011828] .do_IRQ+0x98/0x110
[ 158.043040] [c00000000164f780] [c000000000002258] hardware_interrupt_common+0x158/0x180
[ 158.043047] --- Exception: 501 at .plpar_hcall_norets+0x84/0xd4
[ 158.043047] LR = .check_and_cede_processor+0x2c/0x40
[ 158.043051] [c00000000164fa70] [c000000000083bc8] .check_and_cede_processor+0x18/0x40 (unreliable)
[ 158.043054] [c00000000164fae0] [c000000000083d24] .shared_cede_loop+0x34/0x80
[ 158.043059] [c00000000164fb60] [c0000000007cdb4c] .cpuidle_enter_state+0x6c/0x150
[ 158.043063] [c00000000164fc10] [c0000000007cdd24] .cpuidle_idle_call+0xf4/0x2f0
[ 158.043067] [c00000000164fcc0] [c000000000077628] .pseries_lpar_idle+0x18/0x60
[ 158.043070] [c00000000164fd30] [c00000000001897c] .arch_cpu_idle+0x6c/0x160
[ 158.043074] [c00000000164fdb0] [c000000000106454] .cpu_startup_entry+0x1d4/0x300
[ 158.043078] [c00000000164fe80] [c00000000000cbdc] .rest_init+0x9c/0xb0
[ 158.043082] [c00000000164fef0] [c000000000d43b28] .start_kernel+0x4dc/0x4f8
[ 158.043085] [c00000000164ff90] [c000000000009fe8] .start_here_common+0x20/0x38
[ 158.043102] EEH: Detected PCI bus error on PHB#0-PE#10000
[ 158.043110] EEH: This PCI device has failed 1 times in the last hour
[ 158.043117] EEH: Notify device drivers to shutdown
[ 158.043461] EEH: Collect temporary log
[ 158.044976] EEH: of node=/pci@800000020000019/serial@0
[ 158.045075] EEH: PCI device/vendor: 00f6114f
[ 158.045173] EEH: PCI cmd/status register: 00180142
[ 158.045942] EEH: PCI-E capabilities and status follow:
[ 158.046040] EEH: PCI-E 00: 0001b010
[ 158.046138] EEH: PCI-E 01: 00008240
[ 158.046238] EEH: PCI-E 02: 0010501e
[ 158.046375] EEH: PCI-E 03: 00073c11
[ 158.046503] EEH: PCI-E 04: 10110000
[ 158.046627] EEH: PCI-E 05: 00000000
[ 158.046738] EEH: PCI-E 06: 00000000
[ 158.046850] EEH: PCI-E 07: 00000000
[ 158.046948] EEH: PCI-E 08: 00000000
[ 158.048997] RTAS: event: 21, Type: Platform Error, Severity: 2
[ 158.049002] EEH: Reset without hotplug activity
[ 158.688739] EEH: Frozen PE#10000 detected on PHB#0
[ 158.688752] CPU: 1 PID: 1206 Comm: minicom Not tainted 3.13.0-69-generic #112-Ubuntu
[ 158.688753] Call Trace:
[ 158.688757] [c000000c61def6b0] [c000000000016b00] .show_stack+0x170/0x290 (unreliable)
[ 158.688762] [c000000c61def7a0] [c0000000009787d0] .dump_stack+0x88/0xb4
[ 158.688766] [c000000c61def820] [c0000000000365b0] .eeh_dev_check_failure+0x430/0x480
[ 158.688770] [c000000c61def8d0] [c000000000036684] .eeh_check_failure+0x84/0xe0
[ 158.688773] [c000000c61def960] [c0000000005b3c14] .mem_serial_in+0x94/0xc0
[ 158.688777] [c000000c61def9e0] [c0000000005b1b54] .serial8250_modem_status+0x44/0x140
[ 158.688780] [c000000c61defa70] [c0000000005b1c70] .serial8250_get_mctrl+0x20/0x80
[ 158.688783] [c000000c61defaf0] [c0000000005abb38] .uart_tiocmget+0x78/0x100
[ 158.688786] [c000000c61defb80] [c000000000583988] .tty_ioctl+0x418/0xeb0
[ 158.688790] [c000000c61defc90] [c000000000273184] .do_vfs_ioctl+0x4d4/0x790
[ 158.688793] [c000000c61defd80] [c000000000273514] .SyS_ioctl+0xd4/0xf0
[ 158.688797] [c000000c61defe30] [c00000000000a158] syscall_exit+0x0/0x98
[ 160.116437] EEH: bus reset 1 failed on PHB#0-PE#10000, rc=0
[ 160.691346] EEH: Frozen PE#10000 detected on PHB#0
[ 160.691359] CPU: 1 PID: 1206 Comm: minicom Not tainted 3.13.0-69-generic #112-Ubuntu
[ 160.691361] Call Trace:
[ 160.691365] [c000000c61def6b0] [c000000000016b00] .show_stack+0x170/0x290 (unreliable)
[ 160.691370] [c000000c61def7a0] [c0000000009787d0] .dump_stack+0x88/0xb4
[ 160.691374] [c000000c61def820] [c0000000000365b0] .eeh_dev_check_failure+0x430/0x480
[ 160.691377] [c000000c61def8d0] [c000000000036684] .eeh_check_failure+0x84/0xe0
[ 160.691381] [c000000c61def960] [c0000000005b3c14] .mem_serial_in+0x94/0xc0
[ 160.691384] [c000000c61def9e0] [c0000000005b1b54] .serial8250_modem_status+0x44/0x140
[ 160.691387] [c000000c61defa70] [c0000000005b1c70] .serial8250_get_mctrl+0x20/0x80
[ 160.691390] [c000000c61defaf0] [c0000000005abb38] .uart_tiocmget+0x78/0x100
[ 160.691393] [c000000c61defb80] [c000000000583988] .tty_ioctl+0x418/0xeb0
[ 160.691396] [c000000c61defc90] [c000000000273184] .do_vfs_ioctl+0x4d4/0x790
[ 160.691399] [c000000c61defd80] [c000000000273514] .SyS_ioctl+0xd4/0xf0
[ 160.691403] [c000000c61defe30] [c00000000000a158] syscall_exit+0x0/0x98
[ 162.186442] EEH: bus reset 2 failed on PHB#0-PE#10000, rc=0
[ 162.693913] EEH: Frozen PE#10000 detected on PHB#0
[ 162.693925] CPU: 1 PID: 1206 Comm: minicom Not tainted 3.13.0-69-generic #112-Ubuntu
[ 162.693927] Call Trace:
[ 162.693931] [c000000c61def6b0] [c000000000016b00] .show_stack+0x170/0x290 (unreliable)
[ 162.693936] [c000000c61def7a0] [c0000000009787d0] .dump_stack+0x88/0xb4
[ 162.693940] [c000000c61def820] [c0000000000365b0] .eeh_dev_check_failure+0x430/0x480
[ 162.693944] [c000000c61def8d0] [c000000000036684] .eeh_check_failure+0x84/0xe0
[ 162.693947] [c000000c61def960] [c0000000005b3c14] .mem_serial_in+0x94/0xc0
[ 162.693950] [c000000c61def9e0] [c0000000005b1b54] .serial8250_modem_status+0x44/0x140
[ 162.693953] [c000000c61defa70] [c0000000005b1c70] .serial8250_get_mctrl+0x20/0x80
[ 162.693956] [c000000c61defaf0] [c0000000005abb38] .uart_tiocmget+0x78/0x100
[ 162.693959] [c000000c61defb80] [c000000000583988] .tty_ioctl+0x418/0xeb0
[ 162.693962] [c000000c61defc90] [c000000000273184] .do_vfs_ioctl+0x4d4/0x790
[ 162.693965] [c000000c61defd80] [c000000000273514] .SyS_ioctl+0xd4/0xf0
[ 162.693969] [c000000c61defe30] [c00000000000a158] syscall_exit+0x0/0x98
[ 164.256445] EEH: bus reset 3 failed on PHB#0-PE#10000, rc=0
[ 164.256459] eeh_handle_normal_event: Cannot reset, err=-1
[ 164.256462] EEH: Unable to recover from failure from PHB#0-PE#10000.
[ 164.256462] Please try reseating or replacing it
[ 164.311597] RTAS: event: 22, Type: Platform Error, Severity: 2
[ 164.311604] pseries_eeh_configure_bridge: Unable to configure bridge PHB#0-PE#10000 (-1)
[ 164.312773] EEH: of node=/pci@800000020000019/serial@0
[ 164.312873] EEH: PCI device/vendor: 00f6114f
[ 164.312971] EEH: PCI cmd/status register: 00100140
[ 164.313741] EEH: PCI-E capabilities and status follow:
[ 164.313839] EEH: PCI-E 00: 0001b010
[ 164.313937] EEH: PCI-E 01: 00008240
[ 164.314035] EEH: PCI-E 02: 0010501e
[ 164.314133] EEH: PCI-E 03: 00073c11
[ 164.314232] EEH: PCI-E 04: 10110000
[ 164.314329] EEH: PCI-E 05: 00000000
[ 164.314427] EEH: PCI-E 06: 00000000
[ 164.314524] EEH: PCI-E 07: 00000000
[ 164.314621] EEH: PCI-E 08: 00000000
[ 164.316444] RTAS: event: 23, Type: Platform Error, Severity: 2
[ 164.317246] serial 0000:01:00.0: disabling already-disabled device
[ 164.317268] ------------[ cut here ]------------
[ 164.317274] WARNING: at /build/linux-Sf9ehY/linux-3.13.0/drivers/pci/pci.c:1444
[ 164.317278] Modules linked in:
[ 164.317285] CPU: 13 PID: 89 Comm: eehd Not tainted 3.13.0-69-generic #112-Ubuntu
[ 164.317290] task: c000000c64135b60 ti: c000000c64270000 task.ti: c000000c64270000
[ 164.317295] NIP: c000000000520d0c LR: c000000000520d08 CTR: 0000000000000000
[ 164.317300] REGS: c000000c64273480 TRAP: 0700 Not tainted (3.13.0-69-generic)
[ 164.317304] MSR: 8000000100029033 <SF,EE,ME,IR,DR,RI,LE> CR: 24000044 XER: 20000000
[ 164.317319] CFAR: c000000000970954 SOFTE: 1
[ 164.317319] GPR00: c000000000520d08 c000000c64273700 c000000001650a00 0000000000000036
[ 164.317319] GPR04: 0000000000000000 0000000000000000 0000000000000030 0000000000000037
[ 164.317319] GPR08: c0000000014b0a00 0000000000000000 0000000000000000 0000000000000002
[ 164.317319] GPR12: 0000000024000042 c00000000eae2d80 c0000000000ce2a0 c000000c719e4ac0
[ 164.317319] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 164.317319] GPR20: 0000000000000000 0000000000000000 0000000000000000 c000000000c0d948
[ 164.317319] GPR24: c000000000c0d920 0000000000100100 c0000000015170b0 c000000000c0e838
[ 164.317319] GPR28: c000000001517088 c000000c6ae8b800 c000000c6ae8be20 c000000c6ae8b800
[ 164.317381] NIP [c000000000520d0c] .pci_disable_device+0xdc/0xf0
[ 164.317387] LR [c000000000520d08] .pci_disable_device+0xd8/0xf0
[ 164.317390] Call Trace:
[ 164.317396] [c000000c64273700] [c000000000520d08] .pci_disable_device+0xd8/0xf0 (unreliable)
[ 164.317406] [c000000c64273780] [c0000000005bd224] .pciserial_remove_one+0x34/0x50
[ 164.317414] [c000000c64273800] [c0000000005232e4] .pci_device_remove+0x54/0xb0
[ 164.317422] [c000000c64273880] [c0000000005e5d88] .__device_release_driver+0x88/0x110
[ 164.317429] [c000000c64273900] [c0000000005e5e48] .device_release_driver+0x38/0x60
[ 164.317437] [c000000c64273980] [c00000000051b174] .pci_stop_bus_device+0xa4/0xe0
[ 164.317444] [c000000c64273a10] [c00000000051b324] .pci_stop_and_remove_bus_device+0x24/0x40
[ 164.317456] [c000000c64273a90] [c000000000042268] .pcibios_remove_pci_devices+0xa8/0x140
[ 164.317464] [c000000c64273b30] [c000000000039d6c] .eeh_handle_normal_event+0xac/0x3d0
[ 164.317471] [c000000c64273bc0] [c00000000003a0d8] .eeh_handle_event+0x48/0x2f0
[ 164.317478] [c000000c64273c70] [c00000000003a49c] .eeh_event_handler+0x11c/0x1d0
[ 164.317485] [c000000c64273d30] [c0000000000ce3b0] .kthread+0x110/0x130
[ 164.317492] [c000000c64273e30] [c00000000000a460] .ret_from_kernel_thread+0x5c/0x7c
[ 164.317495] Instruction dump:
[ 164.317499] 387f0090 480c02a1 60000000 e8bf00e0 2fa50000 7c641b78 409e0008 e8bf00a0
[ 164.317510] 3c62ff62 3863ebe8 4844fbc5 60000000 <0fe00000> 39200001 3d42ffec 992ada17
[ 164.317521] ---[ end trace 0b19dac5210a183d ]---
[ 164.317606] EEH: Detected PCI bus error on PHB#0-PE#10000
[ 164.317611] eeh_handle_normal_event: Cannot find PCI bus for PHB#0-PE#10000
[ 164.317632] EEH: Detected PCI bus error on PHB#0-PE#10000
[ 164.317636] eeh_handle_normal_event: Cannot find PCI bus for PHB#0-PE#10000
[ 164.317645] EEH: Detected PCI bus error on PHB#0-PE#10000
[ 164.317649] eeh_handle_normal_event: Cannot find PCI bus for PHB#0-PE#10000
[ 164.321118] ------------[ cut here ]------------
[ 164.321123] WARNING: at /build/linux-Sf9ehY/linux-3.13.0/include/linux/kref.h:47
[ 164.321125] Modules linked in:
[ 164.321130] CPU: 13 PID: 1269 Comm: bash Tainted: G W 3.13.0-69-generic #112-Ubuntu
[ 164.321133] task: c000000c623444c0 ti: c000000c62404000 task.ti: c000000c62404000
[ 164.321137] NIP: c0000000004d15f8 LR: c00000000025fe48 CTR: c00000000025fe90
[ 164.321140] REGS: c000000c62407450 TRAP: 0700 Tainted: G W (3.13.0-69-generic)
[ 164.321142] MSR: 8000000000029033 <SF,EE,ME,IR,DR,RI,LE> CR: 24222422 XER: 00000010
[ 164.321151] CFAR: c00000000025fe44 SOFTE: 1
[ 164.321151] GPR00: c00000000025feb4 c000000c624076d0 c000000001650a00 c000000c70a3e068
[ 164.321151] GPR04: c000000c70a3e068 c000000c62407900 0000000000000001 0000000000000000
[ 164.321151] GPR08: 0000000000000000 0000000000000001 c000000001510a00 c000000c5d581bc0
[ 164.321151] GPR12: 0000000024222488 c00000000eae2d80 0000000022000000 0000000010172000
[ 164.321151] GPR16: 0000000000000001 0000000000000000 0000000000000000 c000000c64780320
[ 164.321151] GPR20: c000000c7d050000 0000000000000022 c000000c7d050000 0000000000000000
[ 164.321151] GPR24: 0000000000000020 c000000c62407900 c0000000015c8cc0 c000000c70a3e068
[ 164.321151] GPR28: c000000c6ad23000 0000000000000000 0000000000000000 c000000c70a3e068
[ 164.321189] NIP [c0000000004d15f8] .kobject_get+0x38/0x60
[ 164.321194] LR [c00000000025fe48] .cdev_get+0x48/0x90
[ 164.321196] Call Trace:
[ 164.321199] [c000000c624076d0] [c000000c62407750] 0xc000000c62407750 (unreliable)
[ 164.321204] [c000000c62407750] [c00000000025feb4] .exact_lock+0x24/0x40
[ 164.321208] [c000000c624077d0] [c0000000005eaa64] .kobj_lookup+0x104/0x1f0
[ 164.321212] [c000000c62407890] [c000000000260620] .chrdev_open+0x190/0x270
[ 164.321216] [c000000c62407940] [c000000000256664] .do_dentry_open+0x234/0x3a0
[ 164.321221] [c000000c624079f0] [c00000000026d6a0] .do_last+0x1b0/0xf10
[ 164.321225] [c000000c62407b10] [c00000000026e500] .path_openat+0x100/0x830
[ 164.321230] [c000000c62407c30] [c000000000270170] .do_filp_open+0x50/0xc0
[ 164.321234] [c000000c62407d60] [c000000000258508] .do_sys_open+0x1b8/0x380
[ 164.321238] [c000000c62407e30] [c00000000000a158] syscall_exit+0x0/0x98
[ 164.321240] Instruction dump:
[ 164.321243] 39430038 7c2004ac 7d205028 31290001 7d20512d 40c2fff4 7c0004ac 2f890001
[ 164.321250] 4dfd0020 3d42ffec 892ada0c 69290001 <0b090000> 2fa90000 4dfe0020 39200001
[ 164.321258] ---[ end trace 0b19dac5210a183e ]---
Fix applied to tty-next. Please pick up when possible.
("serial: 8250_pci: Detach low-level driver during PCI error recovery")
https://git.kernel.org/cgit/linux/kernel/git/gregkh/tty.git/commit/?h=tty-next&id=f209fa03fc9d131b3108c2e4936181eabab87416
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1646857/+subscriptions
