kernel-packages team mailing list archive

Thread
Date
[Bug 1424275] [NEW] Ubuntu 15.04 (non-virtualized): System hung after kdump is trigger from Power NV system

To: kernel-packages@xxxxxxxxxxxxxxxxxxx
From: Launchpad Bug Tracker <1424275@xxxxxxxxxxxxxxxxxx>
Date: Sun, 22 Feb 2015 15:58:14 -0000
Reply-to: Bug 1424275 <1424275@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
You have been subscribed to a public bug:

== Comment: #0  ==
---Problem Description---
Ubuntu 15.04 (non-virtualized): System hung after kdump is trigger from Power NV system
 
---uname output---
ubuntu 15.04 kernel
uname -a:
Linux lep8d 3.18.0-12-generic #13-Ubuntu SMP Thu Jan 29 13:44:26 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux

Machine Type = Power 8 NV


Steps to reproduce:
1) root@lep8d:~# cat /proc/cmdline 
	root=UUID=0d90e79d-3d3f-4d8e-9de9-af000b6f285d ro quiet splash crashkernel=4096M

2) update-grub

3) reboot

4) root@lep8d:~# kdump-config show
USE_KDUMP:        1
KDUMP_SYSCTL:     kernel.panic_on_oops=1
KDUMP_COREDIR:    /var/crash
crashkernel addr: 
current state:    ready to kdump

kexec command:
  /sbin/kexec -p --args-linux --command-line="root=UUID=0d90e79d-3d3f-4d8e-9de9-af000b6f285d ro quiet splash irqpoll maxcpus=1 nousb systemd.unit=kdump-tools.service" --initrd=/boot/initrd.img-3.18.0-12-generic /boot/vmlinux-3.18.0-12-generic
root@lep8d:~# 

5) root@lep8d:/var/crash# echo 'c' | sudo tee /proc/sysrq-trigger

[  300.997178] SysRq : Trigger a crash
[  300.997221] Unable to handle kernel paging request for data at address 0x00000000
[  300.997264] Faulting instruction address: 0xc0000000005f9314
[  300.997328] Oops: Kernel access of bad area, sig: 11 [#1]
[  300.997378] SMP NR_CPUS=2048 NUMA PowerNV
[  300.997430] Modules linked in: uio_pdrv_genirq uio powernv_rng ses enclosure lpfc ipr scsi_transport_fc
[  300.997595] CPU: 64 PID: 2219 Comm: tee Not tainted 3.18.0-12-generic #13-Ubuntu
[  300.997669] task: c000001fdf647e60 ti: c000001fdf698000 task.ti: c000001fdf698000
[  300.997742] NIP: c0000000005f9314 LR: c0000000005fa3ac CTR: c0000000005f92e0
[  300.997816] REGS: c000001fdf69b9d0 TRAP: 0300   Not tainted  (3.18.0-12-generic)
[  300.997889] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE>  CR: 28004022  XER: 20000000
[  300.998075] CFAR: c0000000000084d8 DAR: 0000000000000000 DSISR: 42000000 SOFTE: 1 
GPR00: c0000000005fa3ac c000001fdf69bc50 c000000001426990 0000000000000063 
GPR04: c00000010c0083f8 c00000010c018db0 0000000000000599 c0000000041dfdf0 
GPR08: c000000000e76990 0000000000000001 0000000000000000 0000000000000001 
GPR12: c0000000005f92e0 c000000007b44000 0000000000000000 0000000000000000 
GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 
GPR20: 0000000000000000 0000000000000000 0000000010009dd8 0000000000000001 
GPR24: 0000000010009dd8 00003fffc9d48270 c000000001375708 0000000000000004 
GPR28: 0000000000000063 c00000000133f598 c000000001375ac8 0000000000000000 
[  300.999053] NIP [c0000000005f9314] sysrq_handle_crash+0x34/0x50
[  300.999116] LR [c0000000005fa3ac] __handle_sysrq+0xec/0x280
[  300.999166] Call Trace:
[  300.999192] [c000001fdf69bc50] [c000001fdf69bd90] 0xc000001fdf69bd90 (unreliable)
[  300.999279] [c000001fdf69bc70] [c0000000005fa3ac] __handle_sysrq+0xec/0x280
[  300.999354] [c000001fdf69bd10] [c0000000005faba8] write_sysrq_trigger+0x78/0xa0
[  300.999443] [c000001fdf69bd40] [c000000000345b90] proc_reg_write+0xb0/0x110
[  300.999519] [c000001fdf69bd90] [c0000000002b96ac] vfs_write+0xdc/0x260
[  300.999594] [c000001fdf69bde0] [c0000000002ba24c] SyS_write+0x6c/0x110
[  300.999669] [c000001fdf69be30] [c00000000000927c] syscall_exit+0x0/0x7c
[  300.999768] Instruction dump:
[  300.999839] 3842d6b0 7c0802a6 f8010010 f821ffe1 60000000 60000000 3d22001b 39491bdc 
[  301.000068] 39200001 912a0000 7c0004ac 39400000 <992a0000> 38210020 e8010010 7c0803a6 
[  301.000300] ---[ end trace 93f659e78f1526bc ]---
[  301.000390] 
[  301.000484] Sending IPI to other CPUs
[  301.001675] IPI complete
 -> smp_release_cpus()
spinning_secondaries = 159
 <- smp_release_cpus()
 <- setup_system()
[    1.828216] /build/buildd/linux-3.18.0/drivers/rtc/hctosys.c: unable to open rtc device (rtc0)

starting version 218
[    5.230190] lpfc 0001:03:00.0: 0:1303 Link Up Event x1 received Data: x1 x1 x10 x0 x0 x0 0


root@lep8d:~# dpkg --list | grep -i kdump
ii  kdump-tools                         1:1.5.7-4                    all          scripts and tools for automating kdump (Linux crash dumps)
root@lep8d:~# 

root@lep8d:~# dpkg --list | grep -i linux-crashdump 
ii  linux-crashdump                     3.18.0.12.12                 ppc64el      Linux kernel crashdump setup for the latest generic kernel
root@lep8d:~# 

============================
Steps to reproduce:
1) Installed kdump tools and linux-crashdump
root@lep8d:~# dpkg --list | grep -i dump
ii  kdump-tools                         1:1.5.7-4                    all          scripts and tools for automating kdump (Linux crash dumps)
ii  linux-crashdump                     3.18.0.12.12                 ppc64el      Linux kernel crashdump setup for the latest generic kernel

2) Enable kdump
USE_KDUMP=1
KDUMP_COREDIR="/var/crash/Feb8"

3) root@lep8d:~#  kdump-config status
current state   : ready to kdump
root@lep8d:~# 

4) root@lep8d:~# cat /proc/cmdline 
root=UUID=0d90e79d-3d3f-4d8e-9de9-af000b6f285d ro quiet splash crashkernel=4096M
root@lep8d:~# 

5) root@lep8d:~# kdump-config show
USE_KDUMP:        1
KDUMP_SYSCTL:     kernel.panic_on_oops=1
KDUMP_COREDIR:    /var/crash
crashkernel addr: 
current state:    ready to kdump

kexec command:
  /sbin/kexec -p --args-linux --command-line="root=UUID=0d90e79d-3d3f-4d8e-9de9-af000b6f285d ro quiet splash irqpoll maxcpus=1 nousb systemd.unit=kdump-tools.service" --initrd=/boot/initrd.img-3.18.0-12-generic /boot/vmlinux-3.18.0-12-generic

6) root@lep8d:~# cat /proc/sys/kernel/sysrq
176

7) trigger dump

root@lep8d:~# echo c > /proc/sysrq-trigger
[  702.915680] SysRq : Trigger a crash
[  702.915743] Unable to handle kernel paging request for data at address 0x00000000
[  702.915816] Faulting instruction address: 0xc0000000005f9314
[  702.915879] Oops: Kernel access of bad area, sig: 11 [#1]
[  702.915927] SMP NR_CPUS=2048 NUMA PowerNV
[  702.915978] Modules linked in: uio_pdrv_genirq uio powernv_rng ses enclosure lpfc ipr scsi_transport_fc
[  702.916142] CPU: 12 PID: 2045 Comm: bash Not tainted 3.18.0-12-generic #13-Ubuntu
[  702.916215] task: c000000fdc499030 ti: c000000fdc54c000 task.ti: c000000fdc54c000
[  702.916288] NIP: c0000000005f9314 LR: c0000000005fa3ac CTR: c0000000005f92e0
[  702.916360] REGS: c000000fdc54f9d0 TRAP: 0300   Not tainted  (3.18.0-12-generic)
[  702.916432] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE>  CR: 28242822  XER: 20000000
[  702.916615] CFAR: c0000000000084d8 DAR: 0000000000000000 DSISR: 42000000 SOFTE: 1 
GPR00: c0000000005fa3ac c000000fdc54fc50 c000000001426990 0000000000000063 
GPR04: c00000010b3083f8 c00000010b318db0 0000000000000599 c0000000041dfda0 
GPR08: c000000000e76990 0000000000000001 0000000000000000 0000000000000001 
GPR12: c0000000005f92e0 c000000007b26c00 0000000000000000 0000000022000000 
GPR16: 000000001016d6e8 0000010032868ab8 0000000010143eb8 00000000100c9390 
GPR20: 0000000000000000 000000001017b008 0000000010143d18 0000000000000000 
GPR24: 0000000010156c00 0000000010178868 c000000001375708 0000000000000004 
GPR28: 0000000000000063 c00000000133f598 c000000001375ac8 0000000000000000 
[  702.917579] NIP [c0000000005f9314] sysrq_handle_crash+0x34/0x50
[  702.917640] LR [c0000000005fa3ac] __handle_sysrq+0xec/0x280
[  702.917688] Call Trace:
[  702.917715] [c000000fdc54fc70] [c0000000005fa3ac] __handle_sysrq+0xec/0x280
[  702.917789] [c000000fdc54fd10] [c0000000005faba8] write_sysrq_trigger+0x78/0xa0
[  702.917876] [c000000fdc54fd40] [c000000000345b90] proc_reg_write+0xb0/0x110
[  702.917951] [c000000fdc54fd90] [c0000000002b96ac] vfs_write+0xdc/0x260
[  702.918025] [c000000fdc54fde0] [c0000000002ba24c] SyS_write+0x6c/0x110
[  702.918099] [c000000fdc54fe30] [c00000000000927c] syscall_exit+0x0/0x7c
[  702.918171] Instruction dump:
[  702.918208] 3842d6b0 7c0802a6 f8010010 f821ffe1 60000000 60000000 3d22001b 39491bdc 
[  702.918330] 39200001 912a0000 7c0004ac 39400000 <992a0000> 38210020 e8010010 7c0803a6 
[  702.918455] ---[ end trace de0be178fae811c7 ]---
[  702.918504] 
[  702.918548] Sending IPI to other CPUs
[  702.919695] IPI complete
 -> smp_release_cpus()
spinning_secondaries = 159
 <- smp_release_cpus()
 <- setup_system()
[    1.790146] /build/buildd/linux-3.18.0/drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
starting version 218
[    5.119894] lpfc 0001:03:00.0: 0:1303 Link Up Event x1 received Data: x1 x1 x10 x0 x0 x0 0
[    5.682631] blk_update_request: I/O error, dev sda, sector 0
[    5.682740] Buffer I/O error on dev sda, logical block 0, async page read
[    6.232603] blk_update_request: I/O error, dev sda, sector 0
[    6.232713] Buffer I/O error on dev sda, logical block 0, async page read
[    6.749296] blk_update_request: I/O error, dev sda, sector 0
[    6.749404] Buffer I/O error on dev sda, logical block 0, async page read
[    7.265979] blk_update_request: I/O error, dev sda, sector 0
[    7.266089] Buffer I/O error on dev sda, logical block 0, async page read
[    7.556728] lpfc 0001:03:00.1: 1:1303 Link Up Event x1 received Data: x1 x1 x10 x0 x0 x0 0
[    7.773261] blk_update_request: I/O error, dev sda, sector 0
[    7.773370] Buffer I/O error on dev sda, logical block 0, async page read
[    8.149349] blk_update_request: I/O error, dev sdc, sector 0
[    8.149458] Buffer I/O error on dev sdc, logical block 0, async page read
[    8.282652] blk_update_request: I/O error, dev sda, sector 0
[    8.282760] Buffer I/O error on dev sda, logical block 0, async page read
[    8.666008] blk_update_request: I/O error, dev sdc, sector 0
[    8.666117] Buffer I/O error on dev sdc, logical block 0, async page read
[    8.799327] blk_update_request: I/O error, dev sda, sector 0
[    8.799435] Buffer I/O error on dev sda, logical block 0, async page read
[    9.182675] blk_update_request: I/O error, dev sdc, sector 0
[    9.182783] Buffer I/O error on dev sdc, logical block 0, async page read
[   10.732722] blk_update_request: I/O error, dev sdc, sector 0
[   10.732833] Buffer I/O error on dev sdc, logical block 0, async page read
[   10.866031] blk_update_request: I/O error, dev sda, sector 0
[   10.866138] Buffer I/O error on dev sda, logical block 0, async page read
[   11.249409] blk_update_request: I/O error, dev sdc, sector 0
[   11.249516] Buffer I/O error on dev sdc, logical block 0, async page read
[   11.382717] blk_update_request: I/O error, dev sda, sector 0
[   11.382824] Buffer I/O error on dev sda, logical block 0, async page read
[   11.382900] Dev sda: unable to read RDB block 0
[   11.766089] blk_update_request: I/O error, dev sdc, sector 0
[   11.766197] Buffer I/O error on dev sdc, logical block 0, async page read
[   11.899388] blk_update_request: I/O error, dev sda, sector 0
[   11.899496] Buffer I/O error on dev sda, logical block 0, async page read
[   12.282748] blk_update_request: I/O error, dev sdc, sector 0
[   12.282857] Buffer I/O error on dev sdc, logical block 0, async page read
[   12.416063] blk_update_request: I/O error, dev sda, sector 0
[   12.416171] Buffer I/O error on dev sda, logical block 0, async page read
[   12.799430] blk_update_request: I/O error, dev sdc, sector 0
[   12.799538] Buffer I/O error on dev sdc, logical block 0, async page read
[   12.932746] blk_update_request: I/O error, dev sda, sector 0
[   12.932854] Buffer I/O error on dev sda, logical block 0, async page read
[   13.832806] Dev sdc: unable to read RDB block 0
[   15.899500] blk_update_request: I/O error, dev sdc, sector 0
[   15.899613] Buffer I/O error on dev sdc, logical block 0, async page read
[   16.416167] blk_update_request: I/O error, dev sdc, sector 0
[   16.416275] Buffer I/O error on dev sdc, logical block 0, async page read
[   16.932859] blk_update_request: I/O error, dev sdc, sector 230686592
[   17.449539] blk_update_request: I/O error, dev sdc, sector 230686592
[   17.449648] Buffer I/O error on dev sdc, logical block 1802239, async page read
[   28.268922] sd 0:2:0:0: [sde] Assuming drive cache: write through
[   28.284920] sd 0:2:1:0: [sdf] Assuming drive cache: write through
[   28.405468] scsi 1:0:0:31: Failed to get diagnostic page 0x8000002
[   28.405580] scsi 1:0:0:31: Failed to bind enclosure -19
[   28.412464] scsi 1:0:1:31: Failed to get diagnostic page 0x8000002
[   28.412563] scsi 1:0:1:31: Failed to bind enclosure -19
[   28.419518] scsi 2:0:0:31: Failed to get diagnostic page 0x8000002
[   28.419629] scsi 2:0:0:31: Failed to bind enclosure -19
[   28.425512] scsi 2:0:1:31: Failed to get diagnostic page 0x8000002
[   28.425636] scsi 2:0:1:31: Failed to bind enclosure -19

[-1;-1f[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[   63.081279] reboot: Restarting system
[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbun


System does not get rebooted and console just stay at above screen
message and I need to do reboot from ASMI console to get system booted
up.


After reboot I don't see crash file 

root@lep8d:~# cd /var/crash/Feb8/

root@lep8d:/var/crash/Feb8# ls
197001010530  kexec_cmd

root@lep8d:/var/crash/Feb8/197001010530# ls
dmesg.197001010530  dump.197001010530

root@lep8d:/var/crash/Feb8/197001010530# dmesg | grep -i crash
[    0.000000] Reserving 4096MB of memory at 128MB for crashkernel (System RAM: 196608MB)
[    0.000000] Kernel command line: root=UUID=0d90e79d-3d3f-4d8e-9de9-af000b6f285d ro quiet splash crashkernel=4096M


== Comment: #3  ==

The problem still can be recreate with latest daily ISO builds.

After a dump, system does not reboot automatically. One has to force a
reboot.

** Affects: linux (Ubuntu)
     Importance: Undecided
         Status: New


** Tags: architecture-ppc64le bot-comment bugnameltc-121534 severity-critical targetmilestone-inin---
-- 
Ubuntu 15.04 (non-virtualized): System hung after kdump is trigger from Power NV system
https://bugs.launchpad.net/bugs/1424275
You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.