← Back to team overview

kernel-packages team mailing list archive

[Bug 1353377] Re: ISST-KVM-UBUNTU1410-LE:kidkvm:ubuntu guest crash running IO stress tests.

 

Do happen to know if this crash started with the 3.16.0-6 kernel?  Or
did it also happen with prior kernels?

** Changed in: linux (Ubuntu)
   Importance: Undecided => High

** Tags added: kernel-da-key

** Tags added: ppc64el

** Tags added: utopic

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1353377

Title:
  ISST-KVM-UBUNTU1410-LE:kidkvm:ubuntu guest crash running IO stress
  tests.

Status in “linux” package in Ubuntu:
  Confirmed

Bug description:
  Problem Description
  ---------------------------------
  ubuntu guest crash running IO stress tests.
   
  uname output
  -----------------------------
   uname -a Linux kidg1 3.16.0-6-generic #11-Ubuntu SMP Mon Jul 28 02:00:45 UTC 2014 ppc64le ppc64le ppc64le GNU/Linux
   
  System Hang
  ------------------------
   System hangs/crashed .
   
  Steps to reproduce
  --------------------------------
  1) Used below XML to define the guest

  [root@kidkvm ~]# virsh dumpxml kidg1
  <domain type='kvm' id='28'>
    <name>kidg1</name>
    <uuid>86748b80-a84b-4c70-b1b3-d2a23d88c7ab</uuid>
    <memory unit='KiB'>4194304</memory>
    <currentMemory unit='KiB'>4194304</currentMemory>
    <vcpu placement='static'>4</vcpu>
    <cputune>
      <vcpupin vcpu='0' cpuset='0,8'/>
      <vcpupin vcpu='1' cpuset='24,32'/>
      <vcpupin vcpu='2' cpuset='0,40'/>
    </cputune>
    <numatune>
      <memory mode='strict' nodeset='0'/>
    </numatune>
    <resource>
      <partition>/machine</partition>
    </resource>
    <os>
      <type arch='ppc64' machine='pseries'>hvm</type>
      <boot dev='hd'/>
      <bootmenu enable='yes'/>
    </os>
    <features>
      <acpi/>
      <apic/>
    </features>
    <cpu mode='custom' match='exact'>
      <model fallback='allow'>power8</model>
      <topology sockets='1' cores='2' threads='2'/>
    </cpu>
    <clock offset='utc'/>
    <on_poweroff>destroy</on_poweroff>
    <on_reboot>restart</on_reboot>
    <on_crash>restart</on_crash>
    <devices>
      <emulator>/usr/bin/qemu-system-ppc64</emulator>
      <disk type='file' device='disk'>
        <driver name='qemu' type='qcow2' cache='none'/>
        <source file='/kidg1_GTOdisk/kidg1-1a.img'/>
        <target dev='sda' bus='scsi'/>
        <alias name='scsi0-0-0-0'/>
        <address type='drive' controller='0' bus='0' target='0' unit='0'/>
      </disk>
      <disk type='file' device='disk'>
        <driver name='qemu' type='qcow2' cache='none'/>
        <source file='/kidg1_SRAID1/kidg1-1SR.img'/>
        <target dev='sdb' bus='scsi'/>
        <alias name='scsi0-0-0-1'/>
        <address type='drive' controller='0' bus='0' target='0' unit='1'/>
      </disk>
      <disk type='file' device='disk'>
        <driver name='qemu' type='qcow2' cache='none'/>
        <source file='/kidg1_SRAID1/kidg1-2SR.img'/>
        <target dev='vda' bus='virtio'/>
        <alias name='virtio-disk0'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
      </disk>
      <controller type='usb' index='0'>
        <alias name='usb0'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
      </controller>
      <controller type='pci' index='0' model='pci-root'>
        <alias name='pci.0'/>
      </controller>
      <controller type='scsi' index='0'>
        <alias name='scsi0'/>
        <address type='spapr-vio' reg='0x2000'/>
      </controller>
      <interface type='bridge'>
        <mac address='52:54:00:f2:d3:e2'/>
        <source bridge='brenP3p5s0f0'/>
        <target dev='vnet3'/>
        <model type='virtio'/>
        <driver name='qemu'/>
        <alias name='net0'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
      </interface>
      <serial type='pty'>
        <source path='/dev/pts/1'/>
        <target port='0'/>
        <alias name='serial0'/>
        <address type='spapr-vio' reg='0x30000000'/>
      </serial>
      <console type='pty' tty='/dev/pts/1'>
        <source path='/dev/pts/1'/>
        <target type='serial' port='0'/>
        <alias name='serial0'/>
        <address type='spapr-vio' reg='0x30000000'/>
      </console>
      <video>
        <model type='vga' vram='9216' heads='1'/>
        <alias name='video0'/>
      </video>
      <memballoon model='none'>
        <alias name='balloon0'/>
      </memballoon>
    </devices>
    <seclabel type='none' model='selinux'/>
  </domain>

  2) Defined guest xml and started console to start the installation.

  3) Installation succeded.

  4) Guest is up and running as below

  [root@magkvm ~]# virsh list --all
   Id    Name                           State
  ----------------------------------------------------
   26     kidg1                        running

  
  5) Initiated IO and BASE stress tests using LE scripts 

  6) Stress tests running

  root@kidg1:~# slj
  STAF running status:
  Job ID Job Name Start Date-Time
  ------ -------- -----------------
  1      IO       20140731-21:59:41
  2      BASE     20140731-21:59:58

  
  7) After 72 hours , tried to login kidg1 guest and it fails as below

  [root@kidkvm ~]# ssh root@kidg1
  ssh: connect to host kidg1 port 22: No route to host
  [root@kidkvm ~]

  8) Verified ping test and it failed
  [root@kidkvm ~]# ping kidg1
  PING kidg1.isst.aus.stglabs.ibm.com (10.33.4.124) 56(84) bytes of data.
  From kidkvm.isst.aus.stglabs.ibm.com (10.33.18.47) icmp_seq=1 Destination Host U nreachable
  From kidkvm.isst.aus.stglabs.ibm.com (10.33.18.47) icmp_seq=2 Destination Host U nreachable
  From kidkvm.isst.aus.stglabs.ibm.com (10.33.18.47) icmp_seq=5 Destination Host U nreachable
  From kidkvm.isst.aus.stglab

  9) Tried to login the console using below command
  [root@kidkvm ~]# virsh console --force kidg1
  Connected to domain kidg1

  --> It hangs for a long time

  10) Verified logs on kidkvm host and did not see any errors

  11) Restarted the guest, kidg1 by destroying it.

  12) Seen Stack Traces in /var/log/syslog file on kidg1 guest as below

  Aug  1 06:47:09 kidg1 kernel: [69250.533913] zram0: detected capacity change from 536870912 to 0
       13 Aug  1 06:47:43 kidg1 kernel: [69284.609537] ------------[ cut here ]------------
       14 Aug  1 06:47:43 kidg1 kernel: [69284.609802] WARNING: at /build/buildd/linux-3.16.0/block/blk-mq.c:727
       15 Aug  1 06:47:43 kidg1 kernel: [69284.610298] Modules linked in: lz4_compress rpcsec_gss_krb5 nfsv4 nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache bochs_drm ttm drm_kms_helper drm syscopy 

         area sysfillrect pseries_rng sysimgblt rtc_generic ohci_pci [last unloaded: zram]
       16 Aug  1 06:47:43 kidg1 kernel: [69284.611757] CPU: 1 PID: 15 Comm: kworker/1:0H Not tainted 3.16.0-6-generic #11-Ubuntu
       17 Aug  1 06:47:43 kidg1 kernel: [69284.611950] Workqueue: kblockd blk_mq_run_work_fn
       18 Aug  1 06:47:43 kidg1 kernel: [69284.612110] task: c0000000fc9f6830 ti: c0000000fcbc4000 task.ti: c0000000fcbc4000
       19 Aug  1 06:47:43 kidg1 kernel: [69284.612363] NIP: c0000000004acc18 LR: c0000000000c1eec CTR: c0000000004ad220
       20 Aug  1 06:47:43 kidg1 kernel: [69284.612536] REGS: c0000000fcbc7900 TRAP: 0700   Not tainted  (3.16.0-6-generic)
       21 Aug  1 06:47:43 kidg1 kernel: [69284.612725] MSR: 8000000100029033 <SF,EE,ME,IR,DR,RI,LE>  CR: 22002044  XER: 00000000
       22 Aug  1 06:47:43 kidg1 kernel: [69284.613198] CFAR: c0000000004ad258 SOFTE: 1
       23 Aug  1 06:47:43 kidg1 kernel: [69284.613198] GPR00: c0000000000c1eec c0000000fcbc7b80 c000000001346618 c0000000f9e99000
       24 Aug  1 06:47:43 kidg1 kernel: [69284.613198] GPR04: 0000000000000060 0000000000000000 00003aa318b68523 e1fc185ab7c99088
       25 Aug  1 06:47:43 kidg1 kernel: [69284.613198] GPR08: c0000000f9e99188 0000000000000001 0000000000000005 00000000ffffffff
       26 Aug  1 06:47:43 kidg1 kernel: [69284.613198] GPR12: 0000000000002200 c00000000fe40380 c0000000000cbb28 c0000000fe063640
       27 Aug  1 06:47:43 kidg1 kernel: [69284.613198] GPR16: 0000000000000001 c0000000fb1cf300 c000000001758460 c000000001758580
       28 Aug  1 06:47:43 kidg1 kernel: [69284.613198] GPR20: c000000001758198 c000000001758180 c0000000f9ef3e00 0000000000000001
       29 Aug  1 06:47:43 kidg1 kernel: [69284.613198] GPR24: c0000000012ac344 fffffffffffffef7 0000000000000000 c0000000fcbc7be0
       30 Aug  1 06:47:43 kidg1 kernel: [69284.613198] GPR28: c0000000f9e99000 0000000000000000 c0000000fc910900 c0000000f9e99088
       31 Aug  1 06:47:43 kidg1 kernel: [69284.616021] NIP [c0000000004acc18] __blk_mq_run_hw_queue+0x88/0x460
       32 Aug  1 06:47:43 kidg1 kernel: [69284.616188] LR [c0000000000c1eec] process_one_work+0x1ac/0x4c0
       33 Aug  1 06:47:43 kidg1 kernel: [69284.616350] Call Trace:
       34 Aug  1 06:47:43 kidg1 kernel: [69284.616436] [c0000000fcbc7b80] [0000000000000008] 0x8 (unreliable)
       35 Aug  1 06:47:43 kidg1 kernel: [69284.616819] [c0000000fcbc7c50] [c0000000000c1eec] process_one_work+0x1ac/0x4c0
       36 Aug  1 06:47:43 kidg1 kernel: [69284.617297] [c0000000fcbc7ce0] [c0000000000c2780] worker_thread+0x190/0x640
       37 Aug  1 06:47:43 kidg1 kernel: [69284.617585] [c0000000fcbc7d80] [c0000000000cbc30] kthread+0x110/0x130
       38 Aug  1 06:47:43 kidg1 kernel: [69284.617789] [c0000000fcbc7e30] [c00000000000a3e8] ret_from_kernel_thread+0x5c/0x74
       39 Aug  1 06:47:43 kidg1 kernel: [69284.618112] Instruction dump:
       40 Aug  1 06:47:43 kidg1 kernel: [69284.618211] fb610068 391c0188 eadc0298 fb610060 a12d0008 792ad182 552906be 794a1f24
       41 Aug  1 06:47:43 kidg1 kernel: [69284.618541] 7d48502a 7d494c36 552907fe 69290001 <0b090000> e93c0080 792a07e1 40c202e0
       42 Aug  1 06:47:43 kidg1 kernel: [69284.618931] ---[ end trace c84eabc01b1b3dee ]---
       43 Aug  1 06:50:59 kidg1 kernel: [69480.323014] INFO: task irqbalance:904 blocked for more than 120 seconds.
       44 Aug  1 06:50:59 kidg1 kernel: [69480.323223]       Tainted: G        W     3.16.0-6-generic #11-Ubuntu
       45 Aug  1 06:50:59 kidg1 kernel: [69480.323402] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
       46 Aug  1 06:50:59 kidg1 kernel: [69480.323694] irqbalance      D 00003fff883a9e0c     0   904      1 0x00040000
       47 Aug  1 06:50:59 kidg1 kernel: [69480.324060] Call Trace:
       48 Aug  1 06:50:59 kidg1 kernel: [69480.324283] [c000000003a1b640] [0000000000000001] 0x1 (unreliable)
       49 Aug  1 06:50:59 kidg1 kernel: [69480.324430] [c000000003a1b810] [c000000000016188] __switch_to+0x208/0x350
       50 Aug  1 06:50:59 kidg1 kernel: [69480.324813] [c000000003a1b870] [c0000000009b26f0] __schedule+0x320/0x8e0
       51 Aug  1 06:50:59 kidg1 kernel: [69480.324961] --- Exception: 1000b8b0 at 0x1
       52 Aug  1 06:50:59 kidg1 kernel: [69480.324961]     LR = 0x1002de704b0
       53 Aug  1 06:50:59 kidg1 kernel: [69480.325120] [c000000003a1ba90] [c0000000009b3350] schedule_preempt_disabled+0x20/0x30 (unreliable)
       54 Aug  1 06:50:59 kidg1 kernel: [69480.325327] [c000000003a1bab0] [c0000000009b56ac] __mutex_lock_slowpath+0x10c/0x2c0
       55 Aug  1 06:50:59 kidg1 kernel: [69480.325592] [c000000003a1bb30] [c0000000009b58cc] mutex_lock+0x6c/0x70
       56 Aug  1 06:50:59 kidg1 kernel: [69480.325836] [c000000003a1bb60] [c00000000060a8dc] online_show+0x3c/0x90
       57 Aug  1 06:50:59 kidg1 kernel: [69480.325998] [c000000003a1bbe0] [c00000000060b43c] dev_attr_show+0x5c/0xc0
       58 Aug  1 06:50:59 kidg1 kernel: [69480.326161] [c000000003a1bc20] [c0000000003300c4] sysfs_kf_seq_show+0x114/0x200
       59 Aug  1 06:50:59 kidg1 kernel: [69480.326455] [c000000003a1bc70] [c00000000032db04] kernfs_seq_show+0x54/0x70
       60 Aug  1 06:50:59 kidg1 kernel: [69480.326616] [c000000003a1bca0] [c0000000002be51c] seq_read+0xec/0x500
       61 Aug  1 06:50:59 kidg1 kernel: [69480.326778] [c000000003a1bd40] [c00000000032ee94] kernfs_fop_read+0x184/0x210
       62 Aug  1 06:50:59 kidg1 kernel: [69480.326939] [c000000003a1bd90] [c00000000028c0c0] vfs_read+0xc0/0x200
       63 Aug  1 06:50:59 kidg1 kernel: [69480.327119] [c000000003a1bde0] [c00000000028cf6c] SyS_read+0x6c/0x110
       64 Aug  1 06:50:59 kidg1 kernel: [69480.327311] [c000000003a1be30] [c00000000000a0fc] syscall_exit+0x0/0x7c
       65 Aug  1 06:50:59 kidg1 kernel: [69480.327505] INFO: task kworker/2:0:23287 blocked for more than 120 seconds.
       66 Aug  1 06:50:59 kidg1 kernel: [69480.327793]       Tainted: G        W     3.16.0-6-generic #11-Ubuntu
       67 Aug  1 06:50:59 kidg1 kernel: [69480.327972] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
       68 Aug  1 06:50:59 kidg1 kernel: [69480.328125] kworker/2:0     D 0000000000000000     0 23287      2 0x00000800
       69 Aug  1 06:50:59 kidg1 kernel: [69480.328287] Workqueue: events cpuset_hotplug_workfn


  == Comment: #2 - Vaishnavi Bhat <vaish123@xxxxxxxxxx> -  ==
  From the traces looks like the aio-stress test is leading to the traces. 

  4 Aug  1 06:47:43 kidg1 kernel: [69284.609802] WARNING: at /build/buildd/linux-3.16.0/block/blk-mq.c:727
       15 Aug  1 06:47:43 kidg1 kernel: [69284.610298] Modules linked in: lz4_compress rpcsec_gss_krb5 nfsv4 nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache bochs_drm ttm drm_kms_helper drm syscopy        area sysfillrect pseries_rng sysimgblt rtc_generic ohci_pci [last unloaded: zram]
       16 Aug  1 06:47:43 kidg1 kernel: [69284.611757] CPU: 1 PID: 15 Comm: kworker/1:0H Not tainted 3.16.0-6-generic #11-Ubuntu
       17 Aug  1 06:47:43 kidg1 kernel: [69284.611950] Workqueue: kblockd blk_mq_run_work_fn
       18 Aug  1 06:47:43 kidg1 kernel: [69284.612110] task: c0000000fc9f6830 ti: c0000000fcbc4000 task.ti: c0000000fcbc4000
       19 Aug  1 06:47:43 kidg1 kernel: [69284.612363] NIP: c0000000004acc18 LR: c0000000000c1eec CTR: c0000000004ad220
       20 Aug  1 06:47:43 kidg1 kernel: [69284.612536] REGS: c0000000fcbc7900 TRAP: 0700   Not tainted  (3.16.0-6-generic)
       21 Aug  1 06:47:43 kidg1 kernel: [69284.612725] MSR: 8000000100029033 <SF,EE,ME,IR,DR,RI,LE>  CR: 22002044  XER: 00000000
       22 Aug  1 06:47:43 kidg1 kernel: [69284.613198] CFAR: c0000000004ad258 SOFTE: 1
       23 Aug  1 06:47:43 kidg1 kernel: [69284.613198] GPR00: c0000000000c1eec c0000000fcbc7b80 c000000001346618 c0000000f9e99000
       24 Aug  1 06:47:43 kidg1 kernel: [69284.613198] GPR04: 0000000000000060 0000000000000000 00003aa318b68523 e1fc185ab7c99088
       25 Aug  1 06:47:43 kidg1 kernel: [69284.613198] GPR08: c0000000f9e99188 0000000000000001 0000000000000005 00000000ffffffff
       26 Aug  1 06:47:43 kidg1 kernel: [69284.613198] GPR12: 0000000000002200 c00000000fe40380 c0000000000cbb28 c0000000fe063640
       27 Aug  1 06:47:43 kidg1 kernel: [69284.613198] GPR16: 0000000000000001 c0000000fb1cf300 c000000001758460 c000000001758580
       28 Aug  1 06:47:43 kidg1 kernel: [69284.613198] GPR20: c000000001758198 c000000001758180 c0000000f9ef3e00 0000000000000001
       29 Aug  1 06:47:43 kidg1 kernel: [69284.613198] GPR24: c0000000012ac344 fffffffffffffef7 0000000000000000 c0000000fcbc7be0
       30 Aug  1 06:47:43 kidg1 kernel: [69284.613198] GPR28: c0000000f9e99000 0000000000000000 c0000000fc910900 c0000000f9e99088
       31 Aug  1 06:47:43 kidg1 kernel: [69284.616021] NIP [c0000000004acc18] __blk_mq_run_hw_queue+0x88/0x460
       32 Aug  1 06:47:43 kidg1 kernel: [69284.616188] LR [c0000000000c1eec] process_one_work+0x1ac/0x4c0
       33 Aug  1 06:47:43 kidg1 kernel: [69284.616350] Call Trace:
       34 Aug  1 06:47:43 kidg1 kernel: [69284.616436] [c0000000fcbc7b80] [0000000000000008] 0x8 (unreliable)
       35 Aug  1 06:47:43 kidg1 kernel: [69284.616819] [c0000000fcbc7c50] [c0000000000c1eec] process_one_work+0x1ac/0x4c0
       36 Aug  1 06:47:43 kidg1 kernel: [69284.617297] [c0000000fcbc7ce0] [c0000000000c2780] worker_thread+0x190/0x640
       37 Aug  1 06:47:43 kidg1 kernel: [69284.617585] [c0000000fcbc7d80] [c0000000000cbc30] kthread+0x110/0x130
       38 Aug  1 06:47:43 kidg1 kernel: [69284.617789] [c0000000fcbc7e30] [c00000000000a3e8] ret_from_kernel_thread+0x5c/0x74
       39 Aug  1 06:47:43 kidg1 kernel: [69284.618112] Instruction dump:
       40 Aug  1 06:47:43 kidg1 kernel: [69284.618211] fb610068 391c0188 eadc0298 fb610060 a12d0008 792ad182 552906be 794a1f24
       41 Aug  1 06:47:43 kidg1 kernel: [69284.618541] 7d48502a 7d494c36 552907fe 69290001 <0b090000> e93c0080 792a07e1 40c202e0
       42 Aug  1 06:47:43 kidg1 kernel: [69284.618931] ---[ end trace c84eabc01b1b3dee ]---
       43 Aug  1 06:50:59 kidg1 kernel: [69480.323014] INFO: task irqbalance:904 blocked for more than 120 seconds.
       44 Aug  1 06:50:59 kidg1 kernel: [69480.323223]       Tainted: G        W     3.16.0-6-generic #11-Ubuntu
       45 Aug  1 06:50:59 kidg1 kernel: [69480.323402] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
       46 Aug  1 06:50:59 kidg1 kernel: [69480.323694] irqbalance      D 00003fff883a9e0c     0   904      1 0x00040000
       47 Aug  1 06:50:59 kidg1 kernel: [69480.324060] Call Trace:
       48 Aug  1 06:50:59 kidg1 kernel: [69480.324283] [c000000003a1b640] [0000000000000001] 0x1 (unreliable)
       49 Aug  1 06:50:59 kidg1 kernel: [69480.324430] [c000000003a1b810] [c000000000016188] __switch_to+0x208/0x350
       50 Aug  1 06:50:59 kidg1 kernel: [69480.324813] [c000000003a1b870] [c0000000009b26f0] __schedule+0x320/0x8e0
       51 Aug  1 06:50:59 kidg1 kernel: [69480.324961] --- Exception: 1000b8b0 at 0x1
       52 Aug  1 06:50:59 kidg1 kernel: [69480.324961]     LR = 0x1002de704b0
       53 Aug  1 06:50:59 kidg1 kernel: [69480.325120] [c000000003a1ba90] [c0000000009b3350] schedule_preempt_disabled+0x20/0x30 (unreliable)
       54 Aug  1 06:50:59 kidg1 kernel: [69480.325327] [c000000003a1bab0] [c0000000009b56ac] __mutex_lock_slowpath+0x10c/0x2c0
       55 Aug  1 06:50:59 kidg1 kernel: [69480.325592] [c000000003a1bb30] [c0000000009b58cc] mutex_lock+0x6c/0x70
       56 Aug  1 06:50:59 kidg1 kernel: [69480.325836] [c000000003a1bb60] [c00000000060a8dc] online_show+0x3c/0x90
       57 Aug  1 06:50:59 kidg1 kernel: [69480.325998] [c000000003a1bbe0] [c00000000060b43c] dev_attr_show+0x5c/0xc0
       58 Aug  1 06:50:59 kidg1 kernel: [69480.326161] [c000000003a1bc20] [c0000000003300c4] sysfs_kf_seq_show+0x114/0x200
       59 Aug  1 06:50:59 kidg1 kernel: [69480.326455] [c000000003a1bc70] [c00000000032db04] kernfs_seq_show+0x54/0x70
       60 Aug  1 06:50:59 kidg1 kernel: [69480.326616] [c000000003a1bca0] [c0000000002be51c] seq_read+0xec/0x500
       61 Aug  1 06:50:59 kidg1 kernel: [69480.326778] [c000000003a1bd40] [c00000000032ee94] kernfs_fop_read+0x184/0x210
       62 Aug  1 06:50:59 kidg1 kernel: [69480.326939] [c000000003a1bd90] [c00000000028c0c0] vfs_read+0xc0/0x200
       63 Aug  1 06:50:59 kidg1 kernel: [69480.327119] [c000000003a1bde0] [c00000000028cf6c] SyS_read+0x6c/0x110
       64 Aug  1 06:50:59 kidg1 kernel: [69480.327311] [c000000003a1be30] [c00000000000a0fc] syscall_exit+0x0/0x7c
       65 Aug  1 06:50:59 kidg1 kernel: [69480.327505] INFO: task kworker/2:0:23287 blocked for more than 120 seconds.
       66 Aug  1 06:50:59 kidg1 kernel: [69480.327793]       Tainted: G        W     3.16.0-6-generic #11-Ubuntu
       67 Aug  1 06:50:59 kidg1 kernel: [69480.327972] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
       68 Aug  1 06:50:59 kidg1 kernel: [69480.328125] kworker/2:0     D 0000000000000000     0 23287      2 0x00000800
       69 Aug  1 06:50:59 kidg1 kernel: [69480.328287] Workqueue: events cpuset_hotplug_workfn
       70 Aug  1 06:50:59 kidg1 kernel: [69480.329595] Call Trace:
       71 Aug  1 06:50:59 kidg1 kernel: [69480.329656] [c0000000034835a0] [00000001010707c6] 0x1010707c6 (unreliable)
       72 Aug  1 06:50:59 kidg1 kernel: [69480.329827] [c000000003483770] [c000000000016188] __switch_to+0x208/0x350
       73 Aug  1 06:50:59 kidg1 kernel: [69480.330085] [c0000000034837d0] [c0000000009b26f0] __schedule+0x320/0x8e0
       74 Aug  1 06:50:59 kidg1 kernel: [69480.330277] [c0000000034839f0] [c0000000009b3350] schedule_preempt_disabled+0x20/0x30
       75 Aug  1 06:50:59 kidg1 kernel: [69480.330477] [c000000003483a10] [c0000000009b56ac] __mutex_lock_slowpath+0x10c/0x2c0
       76 Aug  1 06:50:59 kidg1 kernel: [69480.330857] [c000000003483a90] [c0000000009b58cc] mutex_lock+0x6c/0x70
       77 Aug  1 06:50:59 kidg1 kernel: [69480.331044] [c000000003483ac0] [c00000000009a080] get_online_cpus+0x50/0x90
       78 Aug  1 06:50:59 kidg1 kernel: [69480.331291] [c000000003483af0] [c00000000015d950] rebuild_sched_domains_locked+0x20/0x80
       79 Aug  1 06:50:59 kidg1 kernel: [69480.331483] [c000000003483b20] [c00000000015ff28] rebuild_sched_domains+0x38/0x60
       80 Aug  1 06:50:59 kidg1 kernel: [69480.331677] [c000000003483b50] [c0000000001604d8] cpuset_hotplug_workfn+0x588/0x680
       81 Aug  1 06:50:59 kidg1 kernel: [69480.331924] [c000000003483c50] [c0000000000c1eec] process_one_work+0x1ac/0x4c0
       82 Aug  1 06:50:59 kidg1 kernel: [69480.332145] [c000000003483ce0] [c0000000000c2780] worker_thread+0x190/0x640
       83 Aug  1 06:50:59 kidg1 kernel: [69480.332390] [c000000003483d80] [c0000000000cbc30] kthread+0x110/0x130
       84 Aug  1 06:50:59 kidg1 kernel: [69480.333312] [c000000003483e30] [c00000000000a3e8] ret_from_kernel_thread+0x5c/0x74
       85 Aug  1 06:50:59 kidg1 kernel: [69480.333508] INFO: task kworker/0:0:3039 blocked for more than 120 seconds.
       86 Aug  1 06:50:59 kidg1 kernel: [69480.333879]       Tainted: G        W     3.16.0-6-generic #11-Ubuntu
       87 Aug  1 06:50:59 kidg1 kernel: [69480.334043] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
       88 Aug  1 06:50:59 kidg1 kernel: [69480.334279] kworker/0:0     D 0000000000000000     0  3039      2 0x00000800
       89 Aug  1 06:50:59 kidg1 kernel: [69480.334462] Workqueue: events rtas_event_scan
       90 Aug  1 06:50:59 kidg1 kernel: [69480.334664] Call Trace:
       91 Aug  1 06:50:59 kidg1 kernel: [69480.334700] [c0000000133b3680] [c0000000133b3890] 0xc0000000133b3890 (unreliable)
       92 Aug  1 06:50:59 kidg1 kernel: [69480.334816] [c0000000133b3850] [c000000000016188] __switch_to+0x208/0x350
       93 Aug  1 06:50:59 kidg1 kernel: [69480.335104] [c0000000133b38b0] [c0000000009b26f0] __schedule+0x320/0x8e0
       94 Aug  1 06:50:59 kidg1 kernel: [69480.335336] [c0000000133b3ad0] [c0000000009b3350] schedule_preempt_disabled+0x20/0x30
       95 Aug  1 06:50:59 kidg1 kernel: [69480.335537] [c0000000133b3af0] [c0000000009b56ac] __mutex_lock_slowpath+0x10c/0x2c0
       96 Aug  1 06:50:59 kidg1 kernel: [69480.335900] [c0000000133b3b70] [c0000000009b58cc] mutex_lock+0x6c/0x70
       97 Aug  1 06:50:59 kidg1 kernel: [69480.336125] [c0000000133b3ba0] [c00000000009a080] get_online_cpus+0x50/0x90
       98 Aug  1 06:50:59 kidg1 kernel: [69480.336301] [c0000000133b3bd0] [c00000000002ebf8] rtas_event_scan+0x98/0x2c0
       99 Aug  1 06:50:59 kidg1 kernel: [69480.336654] [c0000000133b3c50] [c0000000000c1eec] process_one_work+0x1ac/0x4c0
      100 Aug  1 06:50:59 kidg1 kernel: [69480.336761] [c0000000133b3ce0] [c0000000000c2780] worker_thread+0x190/0x640
      101 Aug  1 06:50:59 kidg1 kernel: [69480.336948] [c0000000133b3d80] [c0000000000cbc30] kthread+0x110/0x130
      102 Aug  1 06:50:59 kidg1 kernel: [69480.337124] [c0000000133b3e30] [c00000000000a3e8] ret_from_kernel_thread+0x5c/0x74
      103 Aug  1 06:50:59 kidg1 kernel: [69480.337339] INFO: task aio-stress:19430 blocked for more than 120 seconds.
      104 Aug  1 06:50:59 kidg1 kernel: [69480.338021]       Tainted: G        W     3.16.0-6-generic #11-Ubuntu
      105 Aug  1 06:50:59 kidg1 kernel: [69480.338369] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
      106 Aug  1 06:50:59 kidg1 kernel: [69480.338543] aio-stress      D 00003fff8b270768     0 19430   3891 0x00040000
      107 Aug  1 06:50:59 kidg1 kernel: [69480.338754] Call Trace:
      108 Aug  1 06:50:59 kidg1 kernel: [69480.338817] [c00000001c54f3e0] [c00000001c54f470] 0xc00000001c54f470 (unreliable)
      109 Aug  1 06:50:59 kidg1 kernel: [69480.339062] [c00000001c54f5b0] [c000000000016188] __switch_to+0x208/0x350
      110 Aug  1 06:50:59 kidg1 kernel: [69480.339305] [c00000001c54f610] [c0000000009b26f0] __schedule+0x320/0x8e0
      111 Aug  1 06:50:59 kidg1 kernel: [69480.339646] [c00000001c54f830] [c0000000003a1fd4] jbd2_log_wait_commit+0xc4/0x170
      112 Aug  1 06:50:59 kidg1 kernel: [69480.339863] [c00000001c54f8c0] [c0000000003a4454] __jbd2_journal_force_commit+0x94/0x160
      113 Aug  1 06:50:59 kidg1 kernel: [69480.340226] [c00000001c54f900] [c0000000003a4548] jbd2_journal_force_commit_nested+0x28/0x50
      114 Aug  1 06:50:59 kidg1 kernel: [69480.340469] [c00000001c54f930] [c0000000003344fc] ext4_should_retry_alloc+0x6c/0xb0
      115 Aug  1 06:50:59 kidg1 kernel: [69480.341180] [c00000001c54f960] [c000000000343f14] ext4_write_begin+0x154/0x540
      116 Aug  1 06:50:59 kidg1 kernel: [69480.342812] [c00000001c54fa40] [c0000000001ea4ec] generic_perform_write+0x1ac/0x280
      117 Aug  1 06:50:59 kidg1 kernel: [69480.343429] [c00000001c54fae0] [c0000000001ec88c] __generic_file_write_iter+0x1dc/0x3b0
      118 Aug  1 06:50:59 kidg1 kernel: [69480.344019] [c00000001c54fb40] [c000000000337584] ext4_file_write_iter+0x114/0x400
      119 Aug  1 06:50:59 kidg1 kernel: [69480.344455] [c00000001c54fc00] [c0000000002f20a0] aio_run_iocb+0x270/0x430
      120 Aug  1 06:50:59 kidg1 kernel: [69480.344748] [c00000001c54fcf0] [c0000000002f343c] do_io_submit+0x33c/0x5e0
      121 Aug  1 06:50:59 kidg1 kernel: [69480.344993] [c00000001c54fe30] [c00000000000a0fc] syscall_exit+0x0/0x7c
      122 Aug  1 06:50:59 kidg1 kernel: [69480.345175] INFO: task aio-stress:19431 blocked for more than 120 seconds.
      123 Aug  1 06:50:59 kidg1 kernel: [69480.345457]       Tainted: G        W     3.16.0-6-generic #11-Ubuntu
      124 Aug  1 06:50:59 kidg1 kernel: [69480.345602] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
      125 Aug  1 06:50:59 kidg1 kernel: [69480.345780] aio-stress      D 00003fff8b270768     0 19431   3891 0x00040000
      126 Aug  1 06:50:59 kidg1 kernel: [69480.345974] Call Trace:
      127 Aug  1 06:50:59 kidg1 kernel: [69480.346043] [c00000001c56f3e0] [c00000001c56f470] 0xc00000001c56f470 (unreliable)
      128 Aug  1 06:50:59 kidg1 kernel: [69480.346317] [c00000001c56f5b0] [c000000000016188] __switch_to+0x208/0x350
      129 Aug  1 06:50:59 kidg1 kernel: [69480.347144] [c00000001c56f610] [c0000000009b26f0] __schedule+0x320/0x8e0
      130 Aug  1 06:50:59 kidg1 kernel: [69480.347327] [c00000001c56f830] [c0000000003a1fd4] jbd2_log_wait_commit+0xc4/0x170
      131 Aug  1 06:50:59 kidg1 kernel: [69480.347688] [c00000001c56f8c0] [c0000000003a4454] __jbd2_journal_force_commit+0x94/0x160
      132 Aug  1 06:50:59 kidg1 kernel: [69480.347902] [c00000001c56f900] [c0000000003a4548] jbd2_journal_force_commit_nested+0x28/0x50
      133 Aug  1 06:50:59 kidg1 kernel: [69480.348114] [c00000001c56f930] [c0000000003344fc] ext4_should_retry_alloc+0x6c/0xb0
      134 Aug  1 06:50:59 kidg1 kernel: [69480.348325] [c00000001c56f960] [c000000000343f14] ext4_write_begin+0x154/0x540
      135 Aug  1 06:50:59 kidg1 kernel: [69480.348760] [c00000001c56fa40] [c0000000001ea4ec] generic_perform_write+0x1ac/0x280
      136 Aug  1 06:50:59 kidg1 kernel: [69480.348970] [c00000001c56fae0] [c0000000001ec88c] __generic_file_write_iter+0x1dc/0x3b0
      137 Aug  1 06:50:59 kidg1 kernel: [69480.349185] [c00000001c56fb40] [c000000000337584] ext4_file_write_iter+0x114/0x400
      138 Aug  1 06:50:59 kidg1 kernel: [69480.349397] [c00000001c56fc00] [c0000000002f20a0] aio_run_iocb+0x270/0x430
      139 Aug  1 06:50:59 kidg1 kernel: [69480.349585] [c00000001c56fcf0] [c0000000002f343c] do_io_submit+0x33c/0x5e0
      140 Aug  1 06:50:59 kidg1 kernel: [69480.350124] [c00000001c56fe30] [c00000000000a0fc] syscall_exit+0x0/0x7c
      141 Aug  1 06:50:59 kidg1 kernel: [69480.350794] INFO: task aio-stress:19432 blocked for more than 120 seconds.
      142 Aug  1 06:50:59 kidg1 kernel: [69480.351052]       Tainted: G        W     3.16.0-6-generic #11-Ubuntu
      143 Aug  1 06:50:59 kidg1 kernel: [69480.351371] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
      144 Aug  1 06:50:59 kidg1 kernel: [69480.351717] aio-stress      D 00003fff8b270768     0 19432   3891 0x00040000
      145 Aug  1 06:50:59 kidg1 kernel: [69480.351984] Call Trace:
      146 Aug  1 06:50:59 kidg1 kernel: [69480.352050] [c0000000021b73e0] [c0000000021b7470] 0xc0000000021b7470 (unreliable)
      147 Aug  1 06:50:59 kidg1 kernel: [69480.352594] [c0000000021b75b0] [c000000000016188] __switch_to+0x208/0x350
      148 Aug  1 06:50:59 kidg1 kernel: [69480.352832] [c0000000021b7610] [c0000000009b26f0] __schedule+0x320/0x8e0
      149 Aug  1 06:50:59 kidg1 kernel: [69480.353061] [c0000000021b7830] [c0000000003a1fd4] jbd2_log_wait_commit+0xc4/0x170
      150 Aug  1 06:50:59 kidg1 kernel: [69480.353249] [c0000000021b78c0] [c0000000003a4454] __jbd2_journal_force_commit+0x94/0x160
      151 Aug  1 06:50:59 kidg1 kernel: [69480.353759] [c0000000021b7900] [c0000000003a4548] jbd2_journal_force_commit_nested+0x28/0x50
      152 Aug  1 06:50:59 kidg1 kernel: [69480.354018] [c0000000021b7930] [c0000000003344fc] ext4_should_retry_alloc+0x6c/0xb0
      153 Aug  1 06:50:59 kidg1 kernel: [69480.354199] [c0000000021b7960] [c000000000343f14] ext4_write_begin+0x154/0x540
      154 Aug  1 06:50:59 kidg1 kernel: [69480.354383] [c0000000021b7a40] [c0000000001ea4ec] generic_perform_write+0x1ac/0x280
      155 Aug  1 06:50:59 kidg1 kernel: [69480.354566] [c0000000021b7ae0] [c0000000001ec88c] __generic_file_write_iter+0x1dc/0x3b0
      156 Aug  1 06:50:59 kidg1 kernel: [69480.354828] [c0000000021b7b40] [c000000000337584] ext4_file_write_iter+0x114/0x400
      157 Aug  1 06:50:59 kidg1 kernel: [69480.355083] [c0000000021b7c00] [c0000000002f20a0] aio_run_iocb+0x270/0x430
      158 Aug  1 06:50:59 kidg1 kernel: [69480.355648] [c0000000021b7cf0] [c0000000002f343c] do_io_submit+0x33c/0x5e0
      159 Aug  1 06:50:59 kidg1 kernel: [69480.355814] [c0000000021b7e30] [c00000000000a0fc] syscall_exit+0x0/0x7c
      160 Aug  1 06:50:59 kidg1 kernel: [69480.356066] INFO: task aio-stress:19433 blocked for more than 120 seconds.
      161 Aug  1 06:50:59 kidg1 kernel: [69480.356144]       Tainted: G        W     3.16.0-6-generic #11-Ubuntu
      162 Aug  1 06:50:59 kidg1 kernel: [69480.356239] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
      163 Aug  1 06:50:59 kidg1 kernel: [69480.356475] aio-stress      D 00003fff8b270768     0 19433   3891 0x00040000
      164 Aug  1 06:50:59 kidg1 kernel: [69480.357004] Call Trace:
      165 Aug  1 06:50:59 kidg1 kernel: [69480.357070] [c0000000021873e0] [c000000002187470] 0xc000000002187470 (unreliable)
      166 Aug  1 06:50:59 kidg1 kernel: [69480.357270] [c0000000021875b0] [c000000000016188] __switch_to+0x208/0x350
      167 Aug  1 06:50:59 kidg1 kernel: [69480.357510] [c000000002187610] [c0000000009b26f0] __schedule+0x320/0x8e0
      168 Aug  1 06:50:59 kidg1 kernel: [69480.357830] [c000000002187830] [c0000000003a1fd4] jbd2_log_wait_commit+0xc4/0x170
      169 Aug  1 06:50:59 kidg1 kernel: [69480.358146] [c0000000021878c0] [c0000000003a4454] __jbd2_journal_force_commit+0x94/0x160
      170 Aug  1 06:50:59 kidg1 kernel: [69480.358506] [c000000002187900] [c0000000003a4548] jbd2_journal_force_commit_nested+0x28/0x50
      171 Aug  1 06:50:59 kidg1 kernel: [69480.358853] [c000000002187930] [c0000000003344fc] ext4_should_retry_alloc+0x6c/0xb0
      172 Aug  1 06:50:59 kidg1 kernel: [69480.360057] [c000000002187960] [c000000000343f14] ext4_write_begin+0x154/0x540
      173 Aug  1 06:50:59 kidg1 kernel: [69480.360482] [c000000002187a40] [c0000000001ea4ec] generic_perform_write+0x1ac/0x280
      174 Aug  1 06:50:59 kidg1 kernel: [69480.360779] [c000000002187ae0] [c0000000001ec88c] __generic_file_write_iter+0x1dc/0x3b0
      175 Aug  1 06:50:59 kidg1 kernel: [69480.361158] [c000000002187b40] [c000000000337584] ext4_file_write_iter+0x114/0x400
      176 Aug  1 06:50:59 kidg1 kernel: [69480.361392] [c000000002187c00] [c0000000002f20a0] aio_run_iocb+0x270/0x430
      177 Aug  1 06:50:59 kidg1 kernel: [69480.361726] [c000000002187cf0] [c0000000002f343c] do_io_submit+0x33c/0x5e0
      178 Aug  1 06:50:59 kidg1 kernel: [69480.362004] [c000000002187e30] [c00000000000a0fc] syscall_exit+0x0/0x7c
      179 Aug  1 06:50:59 kidg1 kernel: [69480.362202] INFO: task aio-stress:19434 blocked for more than 120 seconds.
      180 Aug  1 06:50:59 kidg1 kernel: [69480.362308]       Tainted: G        W     3.16.0-6-generic #11-Ubuntu
      181 Aug  1 06:50:59 kidg1 kernel: [69480.362546] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
      182 Aug  1 06:50:59 kidg1 kernel: [69480.362767] aio-stress      D 00003fff8b270768     0 19434   3891 0x00040000
      183 Aug  1 06:50:59 kidg1 kernel: [69480.363132] Call Trace:
      184 Aug  1 06:50:59 kidg1 kernel: [69480.363231] [c00000000218f3e0] [c00000000218f470] 0xc00000000218f470 (unreliable)
      185 Aug  1 06:50:59 kidg1 kernel: [69480.363465] [c00000000218f5b0] [c000000000016188] __switch_to+0x208/0x350
      186 Aug  1 06:50:59 kidg1 kernel: [69480.363807] [c00000000218f610] [c0000000009b26f0] __schedule+0x320/0x8e0
      187 Aug  1 06:50:59 kidg1 kernel: [69480.364067] [c00000000218f830] [c0000000003a1fd4] jbd2_log_wait_commit+0xc4/0x170
      188 Aug  1 06:50:59 kidg1 kernel: [69480.364332] [c00000000218f8c0] [c0000000003a4454] __jbd2_journal_force_commit+0x94/0x160
      189 Aug  1 06:50:59 kidg1 kernel: [69480.364682] [c00000000218f900] [c0000000003a4548] jbd2_journal_force_commit_nested+0x28/0x50
      190 Aug  1 06:50:59 kidg1 kernel: [69480.364944] [c00000000218f930] [c0000000003344fc] ext4_should_retry_alloc+0x6c/0xb0
      191 Aug  1 06:50:59 kidg1 kernel: [69480.365348] [c00000000218f960] [c000000000343f14] ext4_write_begin+0x154/0x540
      192 Aug  1 06:50:59 kidg1 kernel: [69480.365719] [c00000000218fa40] [c0000000001ea4ec] generic_perform_write+0x1ac/0x280
      193 Aug  1 06:50:59 kidg1 kernel: [69480.366261] [c00000000218fae0] [c0000000001ec88c] __generic_file_write_iter+0x1dc/0x3b0
      194 Aug  1 06:50:59 kidg1 kernel: [69480.366650] [c00000000218fb40] [c000000000337584] ext4_file_write_iter+0x114/0x400
      195 Aug  1 06:50:59 kidg1 kernel: [69480.366973] [c00000000218fc00] [c0000000002f20a0] aio_run_iocb+0x270/0x430
      196 Aug  1 06:50:59 kidg1 kernel: [69480.367416] [c00000000218fcf0] [c0000000002f343c] do_io_submit+0x33c/0x5e0
      197 Aug  1 06:50:59 kidg1 kernel: [69480.367718] [c00000000218fe30] [c00000000000a0fc] syscall_exit+0x0/0x7c
      198 Aug  1 06:50:59 kidg1 kernel: [69480.368156] INFO: task aio-stress:19435 blocked for more than 120 seconds.
      199 Aug  1 06:50:59 kidg1 kernel: [69480.368457]       Tainted: G        W     3.16.0-6-generic #11-Ubuntu
      200 Aug  1 06:50:59 kidg1 kernel: [69480.368730] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
      201 Aug  1 06:50:59 kidg1 kernel: [69480.369141] aio-stress      D 00003fff8b270768     0 19435   3891 0x00040000
      202 Aug  1 06:50:59 kidg1 kernel: [69480.369369] Call Trace:
      203 Aug  1 06:50:59 kidg1 kernel: [69480.369480] [c0000000021af3e0] [c0000000021af470] 0xc0000000021af470 (unreliable)
      204 Aug  1 06:50:59 kidg1 kernel: [69480.369885] [c0000000021af5b0] [c000000000016188] __switch_to+0x208/0x350
      205 Aug  1 06:50:59 kidg1 kernel: [69480.370155] [c0000000021af610] [c0000000009b26f0] __schedule+0x320/0x8e0
      206 Aug  1 06:50:59 kidg1 kernel: [69480.370389] [c0000000021af830] [c0000000003a1fd4] jbd2_log_wait_commit+0xc4/0x170
      207 Aug  1 06:50:59 kidg1 kernel: [69480.370648] [c0000000021af8c0] [c0000000003a4454] __jbd2_journal_force_commit+0x94/0x160
      208 Aug  1 06:50:59 kidg1 kernel: [69480.371036] [c0000000021af900] [c0000000003a4548] jbd2_journal_force_commit_nested+0x28/0x50
      209 Aug  1 06:50:59 kidg1 kernel: [69480.371302] [c0000000021af930] [c0000000003344fc] ext4_should_retry_alloc+0x6c/0xb0
      210 Aug  1 06:50:59 kidg1 kernel: [69480.371573] [c0000000021af960] [c000000000343f14] ext4_write_begin+0x154/0x540
      211 Aug  1 06:50:59 kidg1 kernel: [69480.371969] [c0000000021afa40] [c0000000001ea4ec] generic_perform_write+0x1ac/0x280
      212 Aug  1 06:50:59 kidg1 kernel: [69480.372344] [c0000000021afae0] [c0000000001ec88c] __generic_file_write_iter+0x1dc/0x3b0
      213 Aug  1 06:50:59 kidg1 kernel: [69480.372814] [c0000000021afb40] [c000000000337584] ext4_file_write_iter+0x114/0x400
      214 Aug  1 06:50:59 kidg1 kernel: [69480.376761] [c0000000021afc00] [c0000000002f20a0] aio_run_iocb+0x270/0x430
      215 Aug  1 06:50:59 kidg1 kernel: [69480.377036] [c0000000021afcf0] [c0000000002f343c] do_io_submit+0x33c/0x5e0
      216 Aug  1 06:50:59 kidg1 kernel: [69480.377319] [c0000000021afe30] [c00000000000a0fc] syscall_exit+0x0/0x7c
      217 Aug  1 06:50:59 kidg1 kernel: [69480.377601] INFO: task aio-stress:19436 blocked for more than 120 seconds.
      218 Aug  1 06:50:59 kidg1 kernel: [69480.377791]       Tainted: G        W     3.16.0-6-generic #11-Ubuntu
      219 Aug  1 06:50:59 kidg1 kernel: [69480.378225] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
      220 Aug  1 06:50:59 kidg1 kernel: [69480.378475] aio-stress      D 00003fff8b270768     0 19436   3891 0x00040000
      221 Aug  1 06:50:59 kidg1 kernel: [69480.378734] Call Trace:
      222 Aug  1 06:50:59 kidg1 kernel: [69480.378813] [c0000000021ab3e0] [c0000000021ab470] 0xc0000000021ab470 (unreliable)
      223 Aug  1 06:50:59 kidg1 kernel: [69480.379212] [c0000000021ab5b0] [c000000000016188] __switch_to+0x208/0x350
      224 Aug  1 06:50:59 kidg1 kernel: [69480.379451] [c0000000021ab610] [c0000000009b26f0] __schedule+0x320/0x8e0
      225 Aug  1 06:50:59 kidg1 kernel: [69480.379843] [c0000000021ab830] [c0000000003a1fd4] jbd2_log_wait_commit+0xc4/0x170
      226 Aug  1 06:50:59 kidg1 kernel: [69480.380192] [c0000000021ab8c0] [c0000000003a4454] __jbd2_journal_force_commit+0x94/0x160
      227 Aug  1 06:50:59 kidg1 kernel: [69480.380817] [c0000000021ab900] [c0000000003a4548] jbd2_journal_force_commit_nested+0x28/0x50
      228 Aug  1 06:50:59 kidg1 kernel: [69480.381811] [c0000000021ab930] [c0000000003344fc] ext4_should_retry_alloc+0x6c/0xb0
      229 Aug  1 06:50:59 kidg1 kernel: [69480.382151] [c0000000021ab960] [c000000000343f14] ext4_write_begin+0x154/0x540
      230 Aug  1 06:50:59 kidg1 kernel: [69480.382354] [c0000000021aba40] [c0000000001ea4ec] generic_perform_write+0x1ac/0x280
      231 Aug  1 06:50:59 kidg1 kernel: [69480.382706] [c0000000021abae0] [c0000000001ec88c] __generic_file_write_iter+0x1dc/0x3b0
      232 Aug  1 06:50:59 kidg1 kernel: [69480.383070] [c0000000021abb40] [c000000000337584] ext4_file_write_iter+0x114/0x400
      233 Aug  1 06:50:59 kidg1 kernel: [69480.383770] [c0000000021abc00] [c0000000002f20a0] aio_run_iocb+0x270/0x430
      234 Aug  1 06:50:59 kidg1 kernel: [69480.384324] [c0000000021abcf0] [c0000000002f343c] do_io_submit+0x33c/0x5e0
      235 Aug  1 06:50:59 kidg1 kernel: [69480.384796] [c0000000021abe30] [c00000000000a0fc] syscall_exit+0x0/0x7c

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1353377/+subscriptions