kernel-packages team mailing list archive

[Louis Bouchard <louis.bouchard@xxxxxxxxxxxxx>]

Hello,

After investigating the issue, it turns out that an adequate solution is
not possible without sensible modification to the kernel.

As explained before, the kernel allows for the definition of a
vm_free_kbytes which is above the size of the total memory available.

This parameter is enable very early during the boot phase and cannot
easily be disabled.

kdump is not at fault here since it doesn't even get a chance to run
prior to when the OOM starts killing processes.

The only workaround is to remove the definition of vm.vm_free_kbytes
from the /etc/sysctl.conf file so the kernel dump may proceed.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to kexec-tools in Ubuntu.
https://bugs.launchpad.net/bugs/1528101

Title:
  ISST-LTE: kdump failed: second kernel booting hangs after /scripts
  /init-bottom when large min_free_kbytes value being set

Status in kexec-tools package in Ubuntu:
  Invalid
Status in linux package in Ubuntu:
  New

Bug description:
  == Comment: #0 - Ping Tian Han <pthan@xxxxxxxxxx> - 2015-07-15 04:21:23 ==
  ---Problem Description---
  kdump can be triggered by "echo c > /proc/sysrq-trigger', but the second kernel hangs here:

  ...
  [    7.311129] sd 0:2:4:0: alua: rtpg failed with 8070002
  [    7.311232] sd 0:2:4:0: alua: port group 1dc state A preferred supports TOlUSNA
  done.
  Begin: Running /scripts/local-premount ... done.
  [   12.894379] EXT4-fs (dm-12): mounted filesystem with ordered data mode. Opts: (null)
  Begin: Running /scripts/local-bottom ... done.
  done.
  Begin: Running /scripts/init-bottom ... done.
  [   13.463955] init: plymouth-upstart-bridge main process (1681) terminated with status 1
  [   13.463996] init: plymouth-upstart-bridge main process ended, respawning
  [   13.471552] init: plymouth-upstart-bridge main process (1691) terminated with status 1
  [   13.471586] init: plymouth-upstart-bridge main process ended, respawning
  [   13.479547] init: plymouth-upstart-bridge main process (1694) terminated with status 1
  [   13.479580] init: plymouth-upstart-bridge main process ended, respawning
  [   13.487503] init: plymouth-upstart-bridge main process (1696) terminated with status 1
  [   13.487536] init: plymouth-upstart-bridge main process ended, respawning
  [   13.496113] init: plymouth-upstart-bridge main process (1698) terminated with status 1
  [   13.496146] init: plymouth-upstart-bridge main process ended, respawning
  [   13.504363] init: plymouth-upstart-bridge main process (1700) terminated with status 1
  [   13.504397] init: plymouth-upstart-bridge main process ended, respawning
  [   13.512840] init: plymouth-upstart-bridge main process (1702) terminated with status 1
  [   13.512873] init: plymouth-upstart-bridge main process ended, respawning
  [   13.521159] init: plymouth-upstart-bridge main process (1704) terminated with status 1
  [   13.521196] init: plymouth-upstart-bridge main process ended, respawning
  [   13.531934] init: plymouth-upstart-bridge main process (1706) terminated with status 1
  [   13.531968] init: plymouth-upstart-bridge main process ended, respawning
  [   13.548264] init: plymouth-upstart-bridge main process (1708) terminated with status 1
  [   13.548301] init: plymouth-upstart-bridge main process ended, respawning
  [   14.774719] EXT4-fs (dm-12): re-mounted. Opts: errors=remount-ro
  <-----------
  and no vmcore dumpped.
   
  Contact Information = Ping Tian Han/pthan@xxxxxxxxxx, 	Mikhail Afanasiev/afanasie@xxxxxxxxxx 
   
  ---uname output---
  Linux dilllp1 3.19.0-22-generic #22~14.04.1-Ubuntu SMP Wed Jun 17 10:03:39 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux
   
   
  ---System Hang---
   the booting process of the second kernel hangs. We have to reboot the  system.
   
  ---Debugger---
  A debugger is not configured
   
  ---Steps to Reproduce---
   1. install ubuntu on dilllp1, which root device is on a mpath device
  2. sudo apt-get install linux-crashdump
  3. change the crashkerenl= parameter in /boot/grub/grub.cfg to 'crashkernel=768M', then reboot
  4. trigger kdump by 'echo c > /proc/sysrq-trigger'
   
  Userspace tool common name: kdump-tools 
   
  The userspace tool has the following bit modes: 64-bit 

  == Comment: #8 - Ping Tian Han <pthan@xxxxxxxxxx> - 2015-12-09 02:09:30 ==
  We can reproduce this bug with kernel 4.2.0-19.

  == Comment: #12 - Hari Krishna Bathini <hbathini@xxxxxxxxxx> - 2015-12-09 08:22:27 ==
  Is the issue only seen with sysctl configuration
  "vm.min_free_kbytes = 312721" ?

  Thanks
  Hari

  == Comment: #13 - Ping Tian Han <pthan@xxxxxxxxxx> - 2015-12-09 21:08:54 ==
  (In reply to comment #12)
  > Is the issue only seen with sysctl configuration
  > "vm.min_free_kbytes = 312721" ?
  > 

  Yes, I just found that if use the default min_free_kbytes value then
  kdump works just fine!

  == Comment: #14 - Hari Krishna Bathini <hbathini@xxxxxxxxxx> - 2015-12-10 00:51:43 ==
  (In reply to comment #13)
  > (In reply to comment #12)
  > > Is the issue only seen with sysctl configuration
  > > "vm.min_free_kbytes = 312721" ?
  > > 
  > 
  > Yes, I just found that if use the default min_free_kbytes value then kdump
  > works just fine!

  It is blocking kdump from utilising the memory needed to boot.
  So, crashkernel=768M with min_free_kbytes configured to
  312721 translates to ~450 MB for second kernel boot (kdump)
  which doesn't seem to be sufficient for booting.

  How to handle this:

  1. Reserve memory for crashkernel, that augments it by min_free_kbytes

      OR

  2. Use default min_free_kbytes

  Thanks
  Hari

  == Comment: #16 - Ping Tian Han <pthan@xxxxxxxxxx> - 2015-12-10 01:27:40 ==
  (In reply to comment #14)
  > (In reply to comment #13)
  > > (In reply to comment #12)
  > > > Is the issue only seen with sysctl configuration
  > > > "vm.min_free_kbytes = 312721" ?
  > > > 
  > > 
  > > Yes, I just found that if use the default min_free_kbytes value then kdump
  > > works just fine!
  > 
  > It is blocking kdump from utilising the memory needed to boot.
  > So, crashkernel=768M with min_free_kbytes configured to
  > 312721 translates to ~450 MB for second kernel boot (kdump)
  > which doesn't seem to be sufficient for booting.
  > 
  > How to handle this:
  > 
  > 1. Reserve memory for crashkernel, that augments it by min_free_kbytes
  > 
  >     OR
  > 
  > 2. Use default min_free_kbytes
  > 
  > Thanks
  > Hari

  Is it possible that let kdump ignoreis the min_free_kbytes when being
  triggered? We need a large min_free_kbytes to run stress tests, but
  kdump doesn't need it.

  == Comment: #19 - Hari Krishna Bathini <hbathini@xxxxxxxxxx> - 2015-12-20 13:44:12 ==
  Ubuntu uses default initrd and kernel image to boot into kdump
  as well. So, the same sysctl settings (runtime parameters)
  apply for kdump as well. It is difficult to have one sysctl
  setting for production kernel and another for kdump kernel
  for this reason.

  ---
  You can workaround this by using a non-persistent sysctl
  setting in production kernel for vm.min_free_kbytes.

  I mean, remove "vm.min_free_kbytes = 312721" from sysctl
  configuration files and set it with the below command:

          # sudo sysctl vm.min_free_kbytes=312721

  This way, kdump gets to use the default vm.min_free_kbytes
  setting and production kernel could use the value you set.
  ---

  If you wish to have a different soultion, please mirror the
  bug and see what Ubuntu has to say about this..

  Thanks
  Hari

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/kexec-tools/+bug/1528101/+subscriptions