kernel-packages team mailing list archive

Thread
Date
[Bug 1424275] Re: Ubuntu 15.04 (non-virtualized): System hung after kdump is trigger from Power NV system

To: kernel-packages@xxxxxxxxxxxxxxxxxxx
From: "Christopher M. Penalver" <christopher.m.penalver@xxxxxxxxx>
Date: Thu, 05 Mar 2015 09:49:52 -0000
Reply-to: Bug 1424275 <1424275@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: linux (Ubuntu)
   Importance: Undecided => High

** Changed in: linux (Ubuntu)
       Status: New => Triaged

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1424275

Title:
  Ubuntu 15.04 (non-virtualized): System hung after kdump is trigger
  from Power NV system

Status in linux package in Ubuntu:
  Triaged

Bug description:
  == Comment: #0  ==
  ---Problem Description---
  Ubuntu 15.04 (non-virtualized): System hung after kdump is trigger from Power NV system
   
  ---uname output---
  ubuntu 15.04 kernel
  uname -a:
  Linux lep8d 3.18.0-12-generic #13-Ubuntu SMP Thu Jan 29 13:44:26 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux

  Machine Type = Power 8 NV

  
  Steps to reproduce:
  1) root@lep8d:~# cat /proc/cmdline 
  	root=UUID=0d90e79d-3d3f-4d8e-9de9-af000b6f285d ro quiet splash crashkernel=4096M

  2) update-grub

  3) reboot

  4) root@lep8d:~# kdump-config show
  USE_KDUMP:        1
  KDUMP_SYSCTL:     kernel.panic_on_oops=1
  KDUMP_COREDIR:    /var/crash
  crashkernel addr: 
  current state:    ready to kdump

  kexec command:
    /sbin/kexec -p --args-linux --command-line="root=UUID=0d90e79d-3d3f-4d8e-9de9-af000b6f285d ro quiet splash irqpoll maxcpus=1 nousb systemd.unit=kdump-tools.service" --initrd=/boot/initrd.img-3.18.0-12-generic /boot/vmlinux-3.18.0-12-generic
  root@lep8d:~# 

  5) root@lep8d:/var/crash# echo 'c' | sudo tee /proc/sysrq-trigger

  [  300.997178] SysRq : Trigger a crash
  [  300.997221] Unable to handle kernel paging request for data at address 0x00000000
  [  300.997264] Faulting instruction address: 0xc0000000005f9314
  [  300.997328] Oops: Kernel access of bad area, sig: 11 [#1]
  [  300.997378] SMP NR_CPUS=2048 NUMA PowerNV
  [  300.997430] Modules linked in: uio_pdrv_genirq uio powernv_rng ses enclosure lpfc ipr scsi_transport_fc
  [  300.997595] CPU: 64 PID: 2219 Comm: tee Not tainted 3.18.0-12-generic #13-Ubuntu
  [  300.997669] task: c000001fdf647e60 ti: c000001fdf698000 task.ti: c000001fdf698000
  [  300.997742] NIP: c0000000005f9314 LR: c0000000005fa3ac CTR: c0000000005f92e0
  [  300.997816] REGS: c000001fdf69b9d0 TRAP: 0300   Not tainted  (3.18.0-12-generic)
  [  300.997889] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE>  CR: 28004022  XER: 20000000
  [  300.998075] CFAR: c0000000000084d8 DAR: 0000000000000000 DSISR: 42000000 SOFTE: 1 
  GPR00: c0000000005fa3ac c000001fdf69bc50 c000000001426990 0000000000000063 
  GPR04: c00000010c0083f8 c00000010c018db0 0000000000000599 c0000000041dfdf0 
  GPR08: c000000000e76990 0000000000000001 0000000000000000 0000000000000001 
  GPR12: c0000000005f92e0 c000000007b44000 0000000000000000 0000000000000000 
  GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 
  GPR20: 0000000000000000 0000000000000000 0000000010009dd8 0000000000000001 
  GPR24: 0000000010009dd8 00003fffc9d48270 c000000001375708 0000000000000004 
  GPR28: 0000000000000063 c00000000133f598 c000000001375ac8 0000000000000000 
  [  300.999053] NIP [c0000000005f9314] sysrq_handle_crash+0x34/0x50
  [  300.999116] LR [c0000000005fa3ac] __handle_sysrq+0xec/0x280
  [  300.999166] Call Trace:
  [  300.999192] [c000001fdf69bc50] [c000001fdf69bd90] 0xc000001fdf69bd90 (unreliable)
  [  300.999279] [c000001fdf69bc70] [c0000000005fa3ac] __handle_sysrq+0xec/0x280
  [  300.999354] [c000001fdf69bd10] [c0000000005faba8] write_sysrq_trigger+0x78/0xa0
  [  300.999443] [c000001fdf69bd40] [c000000000345b90] proc_reg_write+0xb0/0x110
  [  300.999519] [c000001fdf69bd90] [c0000000002b96ac] vfs_write+0xdc/0x260
  [  300.999594] [c000001fdf69bde0] [c0000000002ba24c] SyS_write+0x6c/0x110
  [  300.999669] [c000001fdf69be30] [c00000000000927c] syscall_exit+0x0/0x7c
  [  300.999768] Instruction dump:
  [  300.999839] 3842d6b0 7c0802a6 f8010010 f821ffe1 60000000 60000000 3d22001b 39491bdc 
  [  301.000068] 39200001 912a0000 7c0004ac 39400000 <992a0000> 38210020 e8010010 7c0803a6 
  [  301.000300] ---[ end trace 93f659e78f1526bc ]---
  [  301.000390] 
  [  301.000484] Sending IPI to other CPUs
  [  301.001675] IPI complete
   -> smp_release_cpus()
  spinning_secondaries = 159
   <- smp_release_cpus()
   <- setup_system()
  [    1.828216] /build/buildd/linux-3.18.0/drivers/rtc/hctosys.c: unable to open rtc device (rtc0)

  starting version 218
  [    5.230190] lpfc 0001:03:00.0: 0:1303 Link Up Event x1 received Data: x1 x1 x10 x0 x0 x0 0

  
  root@lep8d:~# dpkg --list | grep -i kdump
  ii  kdump-tools                         1:1.5.7-4                    all          scripts and tools for automating kdump (Linux crash dumps)
  root@lep8d:~# 

  root@lep8d:~# dpkg --list | grep -i linux-crashdump 
  ii  linux-crashdump                     3.18.0.12.12                 ppc64el      Linux kernel crashdump setup for the latest generic kernel
  root@lep8d:~# 

  ============================
  Steps to reproduce:
  1) Installed kdump tools and linux-crashdump
  root@lep8d:~# dpkg --list | grep -i dump
  ii  kdump-tools                         1:1.5.7-4                    all          scripts and tools for automating kdump (Linux crash dumps)
  ii  linux-crashdump                     3.18.0.12.12                 ppc64el      Linux kernel crashdump setup for the latest generic kernel

  2) Enable kdump
  USE_KDUMP=1
  KDUMP_COREDIR="/var/crash/Feb8"

  3) root@lep8d:~#  kdump-config status
  current state   : ready to kdump
  root@lep8d:~# 

  4) root@lep8d:~# cat /proc/cmdline 
  root=UUID=0d90e79d-3d3f-4d8e-9de9-af000b6f285d ro quiet splash crashkernel=4096M
  root@lep8d:~# 

  5) root@lep8d:~# kdump-config show
  USE_KDUMP:        1
  KDUMP_SYSCTL:     kernel.panic_on_oops=1
  KDUMP_COREDIR:    /var/crash
  crashkernel addr: 
  current state:    ready to kdump

  kexec command:
    /sbin/kexec -p --args-linux --command-line="root=UUID=0d90e79d-3d3f-4d8e-9de9-af000b6f285d ro quiet splash irqpoll maxcpus=1 nousb systemd.unit=kdump-tools.service" --initrd=/boot/initrd.img-3.18.0-12-generic /boot/vmlinux-3.18.0-12-generic

  6) root@lep8d:~# cat /proc/sys/kernel/sysrq
  176

  7) trigger dump

  root@lep8d:~# echo c > /proc/sysrq-trigger
  [  702.915680] SysRq : Trigger a crash
  [  702.915743] Unable to handle kernel paging request for data at address 0x00000000
  [  702.915816] Faulting instruction address: 0xc0000000005f9314
  [  702.915879] Oops: Kernel access of bad area, sig: 11 [#1]
  [  702.915927] SMP NR_CPUS=2048 NUMA PowerNV
  [  702.915978] Modules linked in: uio_pdrv_genirq uio powernv_rng ses enclosure lpfc ipr scsi_transport_fc
  [  702.916142] CPU: 12 PID: 2045 Comm: bash Not tainted 3.18.0-12-generic #13-Ubuntu
  [  702.916215] task: c000000fdc499030 ti: c000000fdc54c000 task.ti: c000000fdc54c000
  [  702.916288] NIP: c0000000005f9314 LR: c0000000005fa3ac CTR: c0000000005f92e0
  [  702.916360] REGS: c000000fdc54f9d0 TRAP: 0300   Not tainted  (3.18.0-12-generic)
  [  702.916432] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE>  CR: 28242822  XER: 20000000
  [  702.916615] CFAR: c0000000000084d8 DAR: 0000000000000000 DSISR: 42000000 SOFTE: 1 
  GPR00: c0000000005fa3ac c000000fdc54fc50 c000000001426990 0000000000000063 
  GPR04: c00000010b3083f8 c00000010b318db0 0000000000000599 c0000000041dfda0 
  GPR08: c000000000e76990 0000000000000001 0000000000000000 0000000000000001 
  GPR12: c0000000005f92e0 c000000007b26c00 0000000000000000 0000000022000000 
  GPR16: 000000001016d6e8 0000010032868ab8 0000000010143eb8 00000000100c9390 
  GPR20: 0000000000000000 000000001017b008 0000000010143d18 0000000000000000 
  GPR24: 0000000010156c00 0000000010178868 c000000001375708 0000000000000004 
  GPR28: 0000000000000063 c00000000133f598 c000000001375ac8 0000000000000000 
  [  702.917579] NIP [c0000000005f9314] sysrq_handle_crash+0x34/0x50
  [  702.917640] LR [c0000000005fa3ac] __handle_sysrq+0xec/0x280
  [  702.917688] Call Trace:
  [  702.917715] [c000000fdc54fc70] [c0000000005fa3ac] __handle_sysrq+0xec/0x280
  [  702.917789] [c000000fdc54fd10] [c0000000005faba8] write_sysrq_trigger+0x78/0xa0
  [  702.917876] [c000000fdc54fd40] [c000000000345b90] proc_reg_write+0xb0/0x110
  [  702.917951] [c000000fdc54fd90] [c0000000002b96ac] vfs_write+0xdc/0x260
  [  702.918025] [c000000fdc54fde0] [c0000000002ba24c] SyS_write+0x6c/0x110
  [  702.918099] [c000000fdc54fe30] [c00000000000927c] syscall_exit+0x0/0x7c
  [  702.918171] Instruction dump:
  [  702.918208] 3842d6b0 7c0802a6 f8010010 f821ffe1 60000000 60000000 3d22001b 39491bdc 
  [  702.918330] 39200001 912a0000 7c0004ac 39400000 <992a0000> 38210020 e8010010 7c0803a6 
  [  702.918455] ---[ end trace de0be178fae811c7 ]---
  [  702.918504] 
  [  702.918548] Sending IPI to other CPUs
  [  702.919695] IPI complete
   -> smp_release_cpus()
  spinning_secondaries = 159
   <- smp_release_cpus()
   <- setup_system()
  [    1.790146] /build/buildd/linux-3.18.0/drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
  starting version 218
  [    5.119894] lpfc 0001:03:00.0: 0:1303 Link Up Event x1 received Data: x1 x1 x10 x0 x0 x0 0
  [    5.682631] blk_update_request: I/O error, dev sda, sector 0
  [    5.682740] Buffer I/O error on dev sda, logical block 0, async page read
  [    6.232603] blk_update_request: I/O error, dev sda, sector 0
  [    6.232713] Buffer I/O error on dev sda, logical block 0, async page read
  [    6.749296] blk_update_request: I/O error, dev sda, sector 0
  [    6.749404] Buffer I/O error on dev sda, logical block 0, async page read
  [    7.265979] blk_update_request: I/O error, dev sda, sector 0
  [    7.266089] Buffer I/O error on dev sda, logical block 0, async page read
  [    7.556728] lpfc 0001:03:00.1: 1:1303 Link Up Event x1 received Data: x1 x1 x10 x0 x0 x0 0
  [    7.773261] blk_update_request: I/O error, dev sda, sector 0
  [    7.773370] Buffer I/O error on dev sda, logical block 0, async page read
  [    8.149349] blk_update_request: I/O error, dev sdc, sector 0
  [    8.149458] Buffer I/O error on dev sdc, logical block 0, async page read
  [    8.282652] blk_update_request: I/O error, dev sda, sector 0
  [    8.282760] Buffer I/O error on dev sda, logical block 0, async page read
  [    8.666008] blk_update_request: I/O error, dev sdc, sector 0
  [    8.666117] Buffer I/O error on dev sdc, logical block 0, async page read
  [    8.799327] blk_update_request: I/O error, dev sda, sector 0
  [    8.799435] Buffer I/O error on dev sda, logical block 0, async page read
  [    9.182675] blk_update_request: I/O error, dev sdc, sector 0
  [    9.182783] Buffer I/O error on dev sdc, logical block 0, async page read
  [   10.732722] blk_update_request: I/O error, dev sdc, sector 0
  [   10.732833] Buffer I/O error on dev sdc, logical block 0, async page read
  [   10.866031] blk_update_request: I/O error, dev sda, sector 0
  [   10.866138] Buffer I/O error on dev sda, logical block 0, async page read
  [   11.249409] blk_update_request: I/O error, dev sdc, sector 0
  [   11.249516] Buffer I/O error on dev sdc, logical block 0, async page read
  [   11.382717] blk_update_request: I/O error, dev sda, sector 0
  [   11.382824] Buffer I/O error on dev sda, logical block 0, async page read
  [   11.382900] Dev sda: unable to read RDB block 0
  [   11.766089] blk_update_request: I/O error, dev sdc, sector 0
  [   11.766197] Buffer I/O error on dev sdc, logical block 0, async page read
  [   11.899388] blk_update_request: I/O error, dev sda, sector 0
  [   11.899496] Buffer I/O error on dev sda, logical block 0, async page read
  [   12.282748] blk_update_request: I/O error, dev sdc, sector 0
  [   12.282857] Buffer I/O error on dev sdc, logical block 0, async page read
  [   12.416063] blk_update_request: I/O error, dev sda, sector 0
  [   12.416171] Buffer I/O error on dev sda, logical block 0, async page read
  [   12.799430] blk_update_request: I/O error, dev sdc, sector 0
  [   12.799538] Buffer I/O error on dev sdc, logical block 0, async page read
  [   12.932746] blk_update_request: I/O error, dev sda, sector 0
  [   12.932854] Buffer I/O error on dev sda, logical block 0, async page read
  [   13.832806] Dev sdc: unable to read RDB block 0
  [   15.899500] blk_update_request: I/O error, dev sdc, sector 0
  [   15.899613] Buffer I/O error on dev sdc, logical block 0, async page read
  [   16.416167] blk_update_request: I/O error, dev sdc, sector 0
  [   16.416275] Buffer I/O error on dev sdc, logical block 0, async page read
  [   16.932859] blk_update_request: I/O error, dev sdc, sector 230686592
  [   17.449539] blk_update_request: I/O error, dev sdc, sector 230686592
  [   17.449648] Buffer I/O error on dev sdc, logical block 1802239, async page read
  [   28.268922] sd 0:2:0:0: [sde] Assuming drive cache: write through
  [   28.284920] sd 0:2:1:0: [sdf] Assuming drive cache: write through
  [   28.405468] scsi 1:0:0:31: Failed to get diagnostic page 0x8000002
  [   28.405580] scsi 1:0:0:31: Failed to bind enclosure -19
  [   28.412464] scsi 1:0:1:31: Failed to get diagnostic page 0x8000002
  [   28.412563] scsi 1:0:1:31: Failed to bind enclosure -19
  [   28.419518] scsi 2:0:0:31: Failed to get diagnostic page 0x8000002
  [   28.419629] scsi 2:0:0:31: Failed to bind enclosure -19
  [   28.425512] scsi 2:0:1:31: Failed to get diagnostic page 0x8000002
  [   28.425636] scsi 2:0:1:31: Failed to bind enclosure -19

  [-1;-1f[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[   63.081279] reboot: Restarting system
  [-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbuntu 15.04[-1;-1f.  .  .  .[-1;-1fUbun


  System does not get rebooted and console just stay at above screen
  message and I need to do reboot from ASMI console to get system booted
  up.

  
  After reboot I don't see crash file 

  root@lep8d:~# cd /var/crash/Feb8/

  root@lep8d:/var/crash/Feb8# ls
  197001010530  kexec_cmd

  root@lep8d:/var/crash/Feb8/197001010530# ls
  dmesg.197001010530  dump.197001010530

  root@lep8d:/var/crash/Feb8/197001010530# dmesg | grep -i crash
  [    0.000000] Reserving 4096MB of memory at 128MB for crashkernel (System RAM: 196608MB)
  [    0.000000] Kernel command line: root=UUID=0d90e79d-3d3f-4d8e-9de9-af000b6f285d ro quiet splash crashkernel=4096M

  
  == Comment: #3  ==

  The problem still can be recreate with latest daily ISO builds.

  After a dump, system does not reboot automatically. One has to force a
  reboot.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1424275/+subscriptions