← Back to team overview

kernel-packages team mailing list archive

[Bug 1589563] Re: ext4 journaling and swapping to same encrypted SSHD hangs system

 

** Attachment added: "Kernel 4.7-rc2 boot and reproduction log"
   https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1589563/+attachment/4680365/+files/kern.log

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1589563

Title:
  ext4 journaling and swapping to same encrypted SSHD hangs system

Status in linux package in Ubuntu:
  Confirmed

Bug description:
  Short description:
  The system has single SSHD disk with luks/lvm2 encrypted root file system and swap partition. If system starts to swap ext4 journal commit and swap requests hang waiting disk io.

  Relevant bug reports in other places:
  So far I have managed to find only one relevant looking report about very similar problem
  https://bbs.archlinux.org/viewtopic.php?id=193034

  Hardware:
  Acer Aspire E15 E5-571G59EQ
  Intel Broadwell i5-5200U
  4GB ram
  nvidia GeForce 840M (binary driver installed but disabled)
  Seagate ST500LM000-1EJ16 500GB HDD + 8GB SSHD

  
  Known ways to reproduce:
  Compiling templated c++ code using g++ 5 that results to g++ optimization taking huge amount of memory.
  I haven't yet tested if there is any other 

  Reproduction details:
  $ cat main.cpp 
  #include <iostream>

  
  template<unsigned idx>
  unsigned loop()
  {
  	if (idx == 1)
  		return 6;

  	unsigned count = 0;
  	for (unsigned i = 1; i < 7; i++) {
  		count += loop< (idx > 1 ? idx - 1 : 1) >();
  	}
  	return count;
  }

  int main(void)
  {
  	unsigned count, dice[5];

  	std::cout << "6^5: " << 6*6*6*6*6 << "\n";
  	
  	std::cout << "loop all: " << loop<5>() << "\n";
  	std::cout << "loop all two players: " << loop<10>() << "\n";
  	return 0;
  }

  $ gcc --version
  gcc (Ubuntu 5.3.1-14ubuntu2.1) 5.3.1 20160413

  $ g++  -O2 -g  -c -o main.o main.cpp

  Relevant backtraces from sysrq-t (recorded using netconsole):

  1. Watchdog catches jdb2 hang

  [ 2784.570413] INFO: task jbd2/dm-1-8:372 blocked for more than 10 seconds.
  [ 2784.571209]       Tainted: P           OE   4.4.0-22-generic #40-Ubuntu
  [ 2784.571948] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  [ 2784.572606] jbd2/dm-1-8     D ffff880035577ad8     0   372      2 0x00000000
  [ 2784.573294]  ffff880035577ad8 ffff880035577b00 ffff880159472940 ffff8801561e0dc0
  [ 2784.573997]  ffff880035578000 ffff88015ecd6d00 7fffffffffffffff ffffffff81821a00
  [ 2784.574724]  ffff880035577c30 ffff880035577af0 ffffffff81821205 0000000000000000
  [ 2784.575445] Call Trace:
  [ 2784.576124]  [<ffffffff81821a00>] ? bit_wait+0x60/0x60
  [ 2784.576835]  [<ffffffff81821205>] schedule+0x35/0x80
  [ 2784.577546]  [<ffffffff81824325>] schedule_timeout+0x1b5/0x270
  [ 2784.578255]  [<ffffffff813b5cf3>] ? __blk_run_queue+0x33/0x40
  [ 2784.578959]  [<ffffffff813bc3d0>] ? blk_queue_bio+0x3d0/0x3e0
  [ 2784.579635]  [<ffffffff810f574c>] ? ktime_get+0x3c/0xb0
  [ 2784.580328]  [<ffffffff81821a00>] ? bit_wait+0x60/0x60
  [ 2784.581047]  [<ffffffff81820754>] io_schedule_timeout+0xa4/0x110
  [ 2784.581767]  [<ffffffff81821a1b>] bit_wait_io+0x1b/0x70
  [ 2784.582509]  [<ffffffff818215ad>] __wait_on_bit+0x5d/0x90
  [ 2784.583215]  [<ffffffff81821a00>] ? bit_wait+0x60/0x60
  [ 2784.583951]  [<ffffffff81821662>] out_of_line_wait_on_bit+0x82/0xb0
  [ 2784.584687]  [<ffffffff810c3ab0>] ? autoremove_wake_function+0x40/0x40
  [ 2784.585434]  [<ffffffff81243c32>] __wait_on_buffer+0x32/0x40
  [ 2784.586174]  [<ffffffff812eaf6f>] jbd2_journal_commit_transaction+0x10cf/0x1870
  [ 2784.586906]  [<ffffffff810ec5fe>] ? try_to_del_timer_sync+0x5e/0x90
  [ 2784.587622]  [<ffffffff812ef32a>] kjournald2+0xca/0x250
  [ 2784.588339]  [<ffffffff810c3a70>] ? wake_atomic_t_function+0x60/0x60
  [ 2784.589058]  [<ffffffff812ef260>] ? commit_timeout+0x10/0x10
  [ 2784.589769]  [<ffffffff810a0588>] kthread+0xd8/0xf0
  [ 2784.590503]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
  [ 2784.591215]  [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
  [ 2784.591930]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0

  2. kswaped looks like hanged too

  [ 2817.029410] kswapd0         D ffff8800934ab608     0    43      2 0x00000000
  [ 2817.030212]  ffff8800934ab608 ffffffff811eaac6 ffff880093b6b700 ffff8800934a0000
  [ 2817.031032]  ffff8800934ac000 ffff8800934ab640 ffff88015eccdd00 ffff88015eccdd00
  [ 2817.031860]  ffff8800934a0000 ffff8800934ab620 ffffffff81821205 0000000100099a03
  [ 2817.032670] Call Trace:
  [ 2817.033473]  [<ffffffff811eaac6>] ? ___slab_alloc+0x2a6/0x460
  [ 2817.034282]  [<ffffffff81821205>] schedule+0x35/0x80    
  [ 2817.035082]  [<ffffffff81824299>] schedule_timeout+0x129/0x270
  [ 2817.035898]  [<ffffffff810ec370>] ? trace_event_raw_event_tick_stop+0x120/0x120
  [ 2817.036709]  [<ffffffff810f574c>] ? ktime_get+0x3c/0xb0
  [ 2817.037515]  [<ffffffff81820754>] io_schedule_timeout+0xa4/0x110
  [ 2817.038327]  [<ffffffff811900c8>] mempool_alloc+0x148/0x170
  [ 2817.039137]  [<ffffffff810c3a70>] ? wake_atomic_t_function+0x60/0x60
  [ 2817.039964]  [<ffffffff813b20ad>] bio_alloc_bioset+0xbd/0x260
  [ 2817.040774]  [<ffffffff816a33cf>] __split_and_process_bio+0x1ef/0x3e0
  [ 2817.041576]  [<ffffffff816a3629>] dm_make_request+0x69/0xc0
  [ 2817.042377]  [<ffffffff813ba502>] generic_make_request+0xf2/0x1d0
  [ 2817.043180]  [<ffffffff813ba656>] submit_bio+0x76/0x170
  [ 2817.043999]  [<ffffffff811d2548>] __swap_writepage+0x228/0x270
  [ 2817.044799]  [<ffffffff811d7bcf>] ? __frontswap_store+0xdf/0x110
  [ 2817.045589]  [<ffffffff811d25c9>] swap_writepage+0x39/0x70
  [ 2817.046386]  [<ffffffff811a0b6d>] pageout.isra.44+0x16d/0x280
  [ 2817.047179]  [<ffffffff811a31ca>] shrink_page_list+0x3ca/0x7a0
  [ 2817.047990]  [<ffffffff811a3c39>] shrink_inactive_list+0x209/0x520
  [ 2817.048785]  [<ffffffff811a48d3>] shrink_lruvec+0x593/0x750
  [ 2817.049582]  [<ffffffff81098239>] ? __queue_work+0x139/0x3b0
  [ 2817.050373]  [<ffffffff811a4b7f>] shrink_zone+0xef/0x2e0
  [ 2817.051159]  [<ffffffff811a5dfe>] kswapd+0x51e/0x990
  [ 2817.051952]  [<ffffffff811a58e0>] ? mem_cgroup_shrink_node_zone+0x1c0/0x1c0
  [ 2817.052731]  [<ffffffff810a0588>] kthread+0xd8/0xf0
  [ 2817.053512]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
  [ 2817.054299]  [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
  [ 2817.055083]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0

  3. encyptfs-thread and bioset looks like sleeping without work

  [ 2817.076543] ecryptfs-kthrea S ffff880093553e40     0    46      2 0x00000000
  [ 2817.077347]  ffff880093553e40 00000000934a29a0 ffff880159472940 ffff8800934a2940
  [ 2817.078160]  ffff880093554000 ffff880093553e80 ffff8800934a2940 0000000000000000
  [ 2817.078980]  0000000000000000 ffff880093553e58 ffffffff81821205 0000000000000000
  [ 2817.079795] Call Trace:
  [ 2817.080596]  [<ffffffff81821205>] schedule+0x35/0x80
  [ 2817.081407]  [<ffffffff8130b02a>] ecryptfs_threadfn+0x17a/0x1c0
  [ 2817.082212]  [<ffffffff810c3a70>] ? wake_atomic_t_function+0x60/0x60
  [ 2817.083017]  [<ffffffff8130aeb0>] ? ecryptfs_add_global_auth_tok+0xa0/0xa0
  [ 2817.083829]  [<ffffffff810a0588>] kthread+0xd8/0xf0
  [ 2817.084628]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
  [ 2817.085432]  [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
  [ 2817.086227]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0

  4. scsi threads seem to be sleeping (for all cores same)

  [ 2817.390470] scsi_eh_0       S ffff8800357c7e10     0   152      2 0x00000000
  [ 2817.391287]  ffff8800357c7e10 ffff8800357c7e08 ffff880159472940 ffff8800937a5280
  [ 2817.392107]  ffff8800357c8000 ffff8800937a5858 ffff8800937a5858 ffff8800937a5280
  [ 2817.392917]  0000000000000000 ffff8800357c7e28 ffffffff81821205 ffff8800356c9800
  [ 2817.393730] Call Trace:
  [ 2817.394537]  [<ffffffff81821205>] schedule+0x35/0x80
  [ 2817.395339]  [<ffffffff815ad2f7>] scsi_error_handler+0x97/0x8a0
  [ 2817.396142]  [<ffffffff81820b46>] ? __schedule+0x386/0xa10
  [ 2817.396946]  [<ffffffff815ad260>] ? scsi_eh_get_sense+0x250/0x250
  [ 2817.397748]  [<ffffffff810a0588>] kthread+0xd8/0xf0
  [ 2817.398553]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
  [ 2817.399363]  [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
  [ 2817.400170]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
  [ 2817.400977] scsi_tmf_0      S ffff8800357cbe38     0   153      2 0x00000000
  [ 2817.401800]  ffff8800357cbe38 ffff8800357cbe40 ffff880159470dc0 ffff8800937a44c0
  [ 2817.402624]  ffff8800357cc000 ffff8800937a44c0 ffffffff8109a830 ffff8800937e9a80
  [ 2817.403439]  ffff8800356e0000 ffff8800357cbe50 ffffffff81821205 ffff8800356e0030
  [ 2817.404257] Call Trace:
  [ 2817.405064]  [<ffffffff8109a830>] ? worker_thread+0x4c0/0x4c0
  [ 2817.405870]  [<ffffffff81821205>] schedule+0x35/0x80
  [ 2817.406673]  [<ffffffff8109ab75>] rescuer_thread+0x345/0x3d0
  [ 2817.407475]  [<ffffffff8109a830>] ? worker_thread+0x4c0/0x4c0
  [ 2817.408279]  [<ffffffff810a0588>] kthread+0xd8/0xf0
  [ 2817.409083]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
  [ 2817.409889]  [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
  [ 2817.410699]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0 

  5. kworker running dm_crypt work seem to be waiting for something in
  lru shrink

  [ 2817.474917] kworker/u16:3   R  running task        0   160      2 0x00000000
  [ 2817.475748] Workqueue: kcryptd kcryptd_crypt [dm_crypt]
  [ 2817.476566]  ffff880035007538 ffffffff811cc3a6 ffff88015723b700 ffff8800354044c0
  [ 2817.477396]  ffff880035008000 ffff880035007570 ffff88015ec0dd00 ffff88015ec0dd00
  [ 2817.478218]  ffffffffffffffe0 ffff880035007550 ffffffff81821205 0000000100099990
  [ 2817.479052] Call Trace:
  [ 2817.479866]  [<ffffffff811cc3a6>] ? page_referenced+0xb6/0x140
  [ 2817.480694]  [<ffffffff81821205>] schedule+0x35/0x80
  [ 2817.481520]  [<ffffffff81824299>] schedule_timeout+0x129/0x270
  [ 2817.482337]  [<ffffffff810ec370>] ? trace_event_raw_event_tick_stop+0x120/0x120
  [ 2817.483173]  [<ffffffff8182443e>] schedule_timeout_uninterruptible+0x1e/0x20
  [ 2817.484009]  [<ffffffff811aea13>] wait_iff_congested+0xf3/0x180
  [ 2817.484847]  [<ffffffff810c3a70>] ? wake_atomic_t_function+0x60/0x60
  [ 2817.485682]  [<ffffffff811a3f2f>] shrink_inactive_list+0x4ff/0x520
  [ 2817.486513]  [<ffffffff811a48d3>] shrink_lruvec+0x593/0x750
  [ 2817.487332]  [<ffffffff811a4b7f>] shrink_zone+0xef/0x2e0
  [ 2817.488151]  [<ffffffff811a4edc>] do_try_to_free_pages+0x16c/0x3f0
  [ 2817.488978]  [<ffffffff811a522e>] try_to_free_pages+0xce/0x180
  [ 2817.489795]  [<ffffffff81196eda>] __alloc_pages_nodemask+0x64a/0xb60
  [ 2817.490608]  [<ffffffff811ade4c>] ? zone_statistics+0x7c/0xa0
  [ 2817.491423]  [<ffffffff811e091c>] alloc_pages_current+0x8c/0x110
  [ 2817.492237]  [<ffffffff811e99ff>] new_slab+0x28f/0x490
  [ 2817.493050]  [<ffffffff811eaa4b>] ___slab_alloc+0x22b/0x460
  [ 2817.493864]  [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
  [ 2817.494663]  [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
  [ 2817.495466]  [<ffffffff811eaca0>] __slab_alloc+0x20/0x40
  [ 2817.496264]  [<ffffffff811eb21f>] kmem_cache_alloc+0x19f/0x1f0
  [ 2817.497065]  [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
  [ 2817.497858]  [<ffffffff8118ff05>] mempool_alloc_slab+0x15/0x20
  [ 2817.498644]  [<ffffffff8118ffee>] mempool_alloc+0x6e/0x170
  [ 2817.499429]  [<ffffffffc0162092>] ? __ablk_encrypt+0x52/0x70 [ablk_helper]
  [ 2817.500218]  [<ffffffff813b20ad>] bio_alloc_bioset+0xbd/0x260
  [ 2817.501003]  [<ffffffffc0112af4>] kcryptd_crypt+0x104/0x380 [dm_crypt]
  [ 2817.501787]  [<ffffffff8109a052>] process_one_work+0x162/0x480
  [ 2817.502572]  [<ffffffff8109a3bb>] worker_thread+0x4b/0x4c0
  [ 2817.503356]  [<ffffffff8109a370>] ? process_one_work+0x480/0x480
  [ 2817.504135]  [<ffffffff810a0588>] kthread+0xd8/0xf0
  [ 2817.504919]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
  [ 2817.505704]  [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
  [ 2817.506490]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0

  [ 2818.290300] kworker/u16:1   D ffff8801561fb538     0  3339      2 0x00000000
  [ 2818.291008] Workqueue: kcryptd kcryptd_crypt [dm_crypt]
  [ 2818.291734]  ffff8801561fb538 ffffffff811cc3a6 ffff880159472940 ffff880093b82940
  [ 2818.292466]  ffff8801561fc000 ffff8801561fb570 ffff88015eccdd00 ffff88015eccdd00
  [ 2818.293201]  ffffffffffffffe0 ffff8801561fb550 ffffffff81821205 0000000100099b27
  [ 2818.293938] Call Trace:
  [ 2818.294658]  [<ffffffff811cc3a6>] ? page_referenced+0xb6/0x140
  [ 2818.295400]  [<ffffffff81821205>] schedule+0x35/0x80
  [ 2818.296162]  [<ffffffff81824299>] schedule_timeout+0x129/0x270
  [ 2818.296924]  [<ffffffff810ec370>] ? trace_event_raw_event_tick_stop+0x120/0x120
  [ 2818.297700]  [<ffffffff8182443e>] schedule_timeout_uninterruptible+0x1e/0x20
  [ 2818.298479]  [<ffffffff811aea13>] wait_iff_congested+0xf3/0x180
  [ 2818.299264]  [<ffffffff810c3a70>] ? wake_atomic_t_function+0x60/0x60
  [ 2818.300063]  [<ffffffff811a3f2f>] shrink_inactive_list+0x4ff/0x520
  [ 2818.300853]  [<ffffffff811a48d3>] shrink_lruvec+0x593/0x750
  [ 2818.301645]  [<ffffffff811a4b7f>] shrink_zone+0xef/0x2e0
  [ 2818.302437]  [<ffffffff811a4edc>] do_try_to_free_pages+0x16c/0x3f0
  [ 2818.303232]  [<ffffffff811a522e>] try_to_free_pages+0xce/0x180
  [ 2818.304038]  [<ffffffff81196eda>] __alloc_pages_nodemask+0x64a/0xb60
  [ 2818.304831]  [<ffffffff811ade4c>] ? zone_statistics+0x7c/0xa0
  [ 2818.305619]  [<ffffffff811e091c>] alloc_pages_current+0x8c/0x110
  [ 2818.306399]  [<ffffffff811e99ff>] new_slab+0x28f/0x490
  [ 2818.307183]  [<ffffffff811eaa4b>] ___slab_alloc+0x22b/0x460
  [ 2818.307975]  [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
  [ 2818.308751]  [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
  [ 2818.309513]  [<ffffffff811eaca0>] __slab_alloc+0x20/0x40
  [ 2818.310279]  [<ffffffff811eb21f>] kmem_cache_alloc+0x19f/0x1f0
  [ 2818.311049]  [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
  [ 2818.311834]  [<ffffffff8118ff05>] mempool_alloc_slab+0x15/0x20
  [ 2818.312603]  [<ffffffff8118ffee>] mempool_alloc+0x6e/0x170
  [ 2818.313377]  [<ffffffffc0162092>] ? __ablk_encrypt+0x52/0x70 [ablk_helper]
  [ 2818.314150]  [<ffffffff813b20ad>] bio_alloc_bioset+0xbd/0x260
  [ 2818.314925]  [<ffffffffc0112af4>] kcryptd_crypt+0x104/0x380 [dm_crypt]
  [ 2818.315712]  [<ffffffff8109a052>] process_one_work+0x162/0x480
  [ 2818.316484]  [<ffffffff8109a3bb>] worker_thread+0x4b/0x4c0
  [ 2818.317250]  [<ffffffff8109a370>] ? process_one_work+0x480/0x480
  [ 2818.318018]  [<ffffffff810a0588>] kthread+0xd8/0xf0
  [ 2818.318782]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
  [ 2818.319561]  [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
  [ 2818.320333]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0

  [ 2818.321113] kworker/u16:2   D ffff880098653538     0  3388      2 0x00000000
  [ 2818.321909] Workqueue: kcryptd kcryptd_crypt [dm_crypt]
  [ 2818.322700]  ffff880098653538 ffffffff811cc3a6 ffff880159471b80 ffff880093b6b700
  [ 2818.323508]  ffff880098654000 ffff880098653570 ffff88015ec8dd00 ffff88015ec8dd00
  [ 2818.324313]  ffffffffffffffe0 ffff880098653550 ffffffff81821205 0000000100099990
  [ 2818.325116] Call Trace:
  [ 2818.325905]  [<ffffffff811cc3a6>] ? page_referenced+0xb6/0x140
  [ 2818.326705]  [<ffffffff81821205>] schedule+0x35/0x80
  [ 2818.327498]  [<ffffffff81824299>] schedule_timeout+0x129/0x270
  [ 2818.328301]  [<ffffffff810ec370>] ? trace_event_raw_event_tick_stop+0x120/0x120
  [ 2818.329103]  [<ffffffff8182443e>] schedule_timeout_uninterruptible+0x1e/0x20
  [ 2818.329908]  [<ffffffff811aea13>] wait_iff_congested+0xf3/0x180
  [ 2818.330711]  [<ffffffff810c3a70>] ? wake_atomic_t_function+0x60/0x60
  [ 2818.331516]  [<ffffffff811a3f2f>] shrink_inactive_list+0x4ff/0x520
  [ 2818.332332]  [<ffffffff811a48d3>] shrink_lruvec+0x593/0x750
  [ 2818.333128]  [<ffffffff811a4b7f>] shrink_zone+0xef/0x2e0
  [ 2818.333912]  [<ffffffff811a4edc>] do_try_to_free_pages+0x16c/0x3f0
  [ 2818.334698]  [<ffffffff811a522e>] try_to_free_pages+0xce/0x180
  [ 2818.335483]  [<ffffffff81194d44>] ? drain_local_pages+0x24/0x30
  [ 2818.336278]  [<ffffffff81196eda>] __alloc_pages_nodemask+0x64a/0xb60
  [ 2818.337060]  [<ffffffff811ade4c>] ? zone_statistics+0x7c/0xa0
  [ 2818.337844]  [<ffffffff811e091c>] alloc_pages_current+0x8c/0x110
  [ 2818.338628]  [<ffffffff811e99ff>] new_slab+0x28f/0x490
  [ 2818.339407]  [<ffffffff811eaa4b>] ___slab_alloc+0x22b/0x460
  [ 2818.340204]  [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
  [ 2818.340982]  [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
  [ 2818.341756]  [<ffffffff811eaca0>] __slab_alloc+0x20/0x40
  [ 2818.342517]  [<ffffffff811eb21f>] kmem_cache_alloc+0x19f/0x1f0
  [ 2818.343276]  [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
  [ 2818.344044]  [<ffffffff8118ff05>] mempool_alloc_slab+0x15/0x20
  [ 2818.344793]  [<ffffffff8118ffee>] mempool_alloc+0x6e/0x170
  [ 2818.345540]  [<ffffffffc0162092>] ? __ablk_encrypt+0x52/0x70 [ablk_helper]
  [ 2818.346287]  [<ffffffff813b20ad>] bio_alloc_bioset+0xbd/0x260
  [ 2818.347022]  [<ffffffffc0112af4>] kcryptd_crypt+0x104/0x380 [dm_crypt]
  [ 2818.347775]  [<ffffffff8109a052>] process_one_work+0x162/0x480
  [ 2818.348518]  [<ffffffff8109a3bb>] worker_thread+0x4b/0x4c0
  [ 2818.349260]  [<ffffffff8109a370>] ? process_one_work+0x480/0x480
  [ 2818.349990]  [<ffffffff810a0588>] kthread+0xd8/0xf0
  [ 2818.350722]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
  [ 2818.351456]  [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
  [ 2818.352199]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0

   2818.478112] kworker/u16:4   D ffff880094c5b538     0  3966      2 0x00000000
  [ 2818.478856] Workqueue: kcryptd kcryptd_crypt [dm_crypt]
  [ 2818.479610]  ffff880094c5b538 ffffffff811cc3a6 ffff880159470dc0 ffff88015723b700
  [ 2818.480374]  ffff880094c5c000 ffff880094c5b570 ffff88015ec4dd00 ffff88015ec4dd00
  [ 2818.481137]  ffffffffffffffe0 ffff880094c5b550 ffffffff81821205 0000000100099b56
  [ 2818.481901] Call Trace:
  [ 2818.482651]  [<ffffffff811cc3a6>] ? page_referenced+0xb6/0x140
  [ 2818.483413]  [<ffffffff81821205>] schedule+0x35/0x80
  [ 2818.484186]  [<ffffffff81824299>] schedule_timeout+0x129/0x270
  [ 2818.484940]  [<ffffffff810ec370>] ? trace_event_raw_event_tick_stop+0x120/0x120
  [ 2818.485691]  [<ffffffff8182443e>] schedule_timeout_uninterruptible+0x1e/0x20
  [ 2818.486447]  [<ffffffff811aea13>] wait_iff_congested+0xf3/0x180
  [ 2818.487201]  [<ffffffff810c3a70>] ? wake_atomic_t_function+0x60/0x60
  [ 2818.487964]  [<ffffffff811a3f2f>] shrink_inactive_list+0x4ff/0x520
  [ 2818.488712]  [<ffffffff811a48d3>] shrink_lruvec+0x593/0x750
  [ 2818.489468]  [<ffffffff811a4b7f>] shrink_zone+0xef/0x2e0
  [ 2818.490209]  [<ffffffff811a4edc>] do_try_to_free_pages+0x16c/0x3f0
  [ 2818.490951]  [<ffffffff811a522e>] try_to_free_pages+0xce/0x180
  [ 2818.491711]  [<ffffffff81194d44>] ? drain_local_pages+0x24/0x30
  [ 2818.492459]  [<ffffffff81196eda>] __alloc_pages_nodemask+0x64a/0xb60
  [ 2818.493202]  [<ffffffff811ade4c>] ? zone_statistics+0x7c/0xa0
  [ 2818.493944]  [<ffffffff811e091c>] alloc_pages_current+0x8c/0x110
  [ 2818.494696]  [<ffffffff811e99ff>] new_slab+0x28f/0x490
  [ 2818.495448]  [<ffffffff811eaa4b>] ___slab_alloc+0x22b/0x460
  [ 2818.496210]  [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
  [ 2818.496964]  [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
  [ 2818.497715]  [<ffffffff811eaca0>] __slab_alloc+0x20/0x40
  [ 2818.498469]  [<ffffffff811eb21f>] kmem_cache_alloc+0x19f/0x1f0
  [ 2818.499220]  [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
  [ 2818.499973]  [<ffffffff8118ff05>] mempool_alloc_slab+0x15/0x20
  [ 2818.500708]  [<ffffffff8118ffee>] mempool_alloc+0x6e/0x170
  [ 2818.501437]  [<ffffffffc0162092>] ? __ablk_encrypt+0x52/0x70 [ablk_helper]
  [ 2818.502169]  [<ffffffff813b20ad>] bio_alloc_bioset+0xbd/0x260
  [ 2818.502888]  [<ffffffffc0112af4>] kcryptd_crypt+0x104/0x380 [dm_crypt]
  [ 2818.503624]  [<ffffffff8109a052>] process_one_work+0x162/0x480
  [ 2818.504346]  [<ffffffff8109a3bb>] worker_thread+0x4b/0x4c0
  [ 2818.505069]  [<ffffffff8109a370>] ? process_one_work+0x480/0x480
  [ 2818.505794]  [<ffffffff8109a370>] ? process_one_work+0x480/0x480
  [ 2818.506496]  [<ffffffff810a0588>] kthread+0xd8/0xf0
  [ 2818.507205]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
  [ 2818.507930]  [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
  [ 2818.508642]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0


  6. kcryptd seems to be sleeping without work

  [ 2817.579092] kcryptd_io      S ffff8800352ebe38     0   318      2 0x00000000
  [ 2817.579901]  ffff8800352ebe38 ffff8800352ebe40 ffffffff81e11500 ffff880035068dc0
  [ 2817.580712]  ffff8800352ec000 ffff880035068dc0 ffffffff8109a830 ffff880035686a80
  [ 2817.581516]  ffff8800937d9180 ffff8800352ebe50 ffffffff81821205 ffff8800937d91b0
  [ 2817.582314] Call Trace:
  [ 2817.583104]  [<ffffffff8109a830>] ? worker_thread+0x4c0/0x4c0
  [ 2817.583904]  [<ffffffff81821205>] schedule+0x35/0x80
  [ 2817.584699]  [<ffffffff8109ab75>] rescuer_thread+0x345/0x3d0
  [ 2817.585494]  [<ffffffff8109a830>] ? worker_thread+0x4c0/0x4c0
  [ 2817.586284]  [<ffffffff810a0588>] kthread+0xd8/0xf0
  [ 2817.587076]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
  [ 2817.587870]  [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
  [ 2817.588663]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
  [ 2817.589463] kcryptd         S ffff8800352efe38     0   319      2 0x00000000
  [ 2817.590270]  ffff8800352efe38 ffff8800352efe40 ffffffff81e11500 ffff880035069b80
  [ 2817.591078]  ffff8800352f0000 ffff880035069b80 ffffffff8109a830 ffff8800355b9480
  [ 2817.591881]  ffff880156062c00 ffff8800352efe50 ffffffff81821205 ffff880156062c30
  [ 2817.592681] Call Trace:
  [ 2817.593468]  [<ffffffff8109a830>] ? worker_thread+0x4c0/0x4c0
  [ 2817.594250]  [<ffffffff81821205>] schedule+0x35/0x80
  [ 2817.595034]  [<ffffffff8109ab75>] rescuer_thread+0x345/0x3d0
  [ 2817.595828]  [<ffffffff8109a830>] ? worker_thread+0x4c0/0x4c0
  [ 2817.596618]  [<ffffffff810a0588>] kthread+0xd8/0xf0
  [ 2817.597413]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
  [ 2817.598214]  [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
  [ 2817.599014]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0

  7. dmcrypt_write seems like sleeping without work

  [ 2817.599818] dmcrypt_write   S ffff8800350f3de8     0   320      2 0x00000000
  [ 2817.600626]  ffff8800350f3de8 ffff8800350f3da8 ffff880159470dc0 ffff88003506a940
  [ 2817.601435]  ffff8800350f4000 ffff8800350f3e40 ffff8800355b9670 ffff88003506a940
  [ 2817.602237]  ffff8800355b9600 ffff8800350f3e00 ffffffff81821205 ffff8800355b9668
  [ 2817.603038] Call Trace:
  [ 2817.603827]  [<ffffffff81821205>] schedule+0x35/0x80
  [ 2817.604618]  [<ffffffffc0113081>] dmcrypt_write+0xd1/0x1e0 [dm_crypt]
  [ 2817.605412]  [<ffffffff810abe40>] ? wake_up_q+0x70/0x70
  [ 2817.606204]  [<ffffffffc0112fb0>] ? crypt_iv_lmk_gen+0xc0/0xc0 [dm_crypt]
  [ 2817.607008]  [<ffffffffc0112fb0>] ? crypt_iv_lmk_gen+0xc0/0xc0 [dm_crypt]
  [ 2817.607807]  [<ffffffff810a0588>] kthread+0xd8/0xf0
  [ 2817.608604]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
  [ 2817.609404]  [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
  [ 2817.610206]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0

  
  8. systemd journal and logind and irqbalanced is blocked in pagefault handler

  [ 2817.716663] systemd-journal D ffff8800351c7b78     0   424      1 0x00000000
  [ 2817.717463]  ffff8800351c7b78 ffff880094103680 ffffffff81e11500 ffff880035458000
  [ 2817.718269]  ffff8800351c8000 ffff88015ec16d00 7fffffffffffffff ffffffff81821a00
  [ 2817.719077]  ffff8800351c7cd8 ffff8800351c7b90 ffffffff81821205 0000000000000000
  [ 2817.719877] Call Trace:
  [ 2817.720663]  [<ffffffff81821a00>] ? bit_wait+0x60/0x60
  [ 2817.721458]  [<ffffffff81821205>] schedule+0x35/0x80
  [ 2817.722247]  [<ffffffff81824325>] schedule_timeout+0x1b5/0x270
  [ 2817.723049]  [<ffffffff813b5cf3>] ? __blk_run_queue+0x33/0x40
  [ 2817.723852]  [<ffffffff813b5fba>] ? queue_unplugged+0x2a/0xb0
  [ 2817.724651]  [<ffffffff81821a00>] ? bit_wait+0x60/0x60
  [ 2817.725448]  [<ffffffff81820754>] io_schedule_timeout+0xa4/0x110
  [ 2817.726244]  [<ffffffff81821a1b>] bit_wait_io+0x1b/0x70
  [ 2817.727021]  [<ffffffff818215ad>] __wait_on_bit+0x5d/0x90
  [ 2817.727800]  [<ffffffff8118f79d>] wait_on_page_bit_killable+0xcd/0xf0
  [ 2817.728583]  [<ffffffff810c3ab0>] ? autoremove_wake_function+0x40/0x40
  [ 2817.729369]  [<ffffffff8118f844>] __lock_page_or_retry+0x84/0xa0
  [ 2817.730143]  [<ffffffff8118fa4d>] filemap_fault+0x1ed/0x3f0
  [ 2817.730920]  [<ffffffff811bc1f0>] __do_fault+0x50/0xe0
  [ 2817.731697]  [<ffffffff811bfc2b>] handle_mm_fault+0xf8b/0x1820
  [ 2817.732475]  [<ffffffff8125575a>] ? ep_poll+0x21a/0x3d0
  [ 2817.733252]  [<ffffffff8106b537>] __do_page_fault+0x197/0x400
  [ 2817.734035]  [<ffffffff8106b7c2>] do_page_fault+0x22/0x30
  [ 2817.734811]  [<ffffffff81827478>] page_fault+0x28/0x30

  [ 2817.906729] systemd-logind  D ffff880157b6bb78     0   857      1 0x00000000
  [ 2817.907551]  ffff880157b6bb78 ffffffff8118d08e ffff880159470dc0 ffff880095402940
  [ 2817.908386]  ffff880157b6c000 ffff88015ec56d00 7fffffffffffffff ffffffff81821a00
  [ 2817.909207]  ffff880157b6bcd8 ffff880157b6bb90 ffffffff81821205 0000000000000000
  [ 2817.910038] Call Trace:
  [ 2817.910850]  [<ffffffff8118d08e>] ? find_get_entry+0x1e/0xa0
  [ 2817.911666]  [<ffffffff81821a00>] ? bit_wait+0x60/0x60
  [ 2817.912477]  [<ffffffff81821205>] schedule+0x35/0x80
  [ 2817.913289]  [<ffffffff81824325>] schedule_timeout+0x1b5/0x270
  [ 2817.914101]  [<ffffffff811c008d>] ? handle_mm_fault+0x13ed/0x1820
  [ 2817.914919]  [<ffffffff81821a00>] ? bit_wait+0x60/0x60
  [ 2817.915731]  [<ffffffff81820754>] io_schedule_timeout+0xa4/0x110
  [ 2817.916546]  [<ffffffff81821a1b>] bit_wait_io+0x1b/0x70
  [ 2817.917353]  [<ffffffff818215ad>] __wait_on_bit+0x5d/0x90
  [ 2817.918151]  [<ffffffff8118f79d>] wait_on_page_bit_killable+0xcd/0xf0
  [ 2817.918952]  [<ffffffff810c3ab0>] ? autoremove_wake_function+0x40/0x40
  [ 2817.919753]  [<ffffffff8118f844>] __lock_page_or_retry+0x84/0xa0
  [ 2817.920557]  [<ffffffff8118fa4d>] filemap_fault+0x1ed/0x3f0
  [ 2817.921349]  [<ffffffff811bc1f0>] __do_fault+0x50/0xe0
  [ 2817.922136]  [<ffffffff811bfc2b>] handle_mm_fault+0xf8b/0x1820
  [ 2817.922924]  [<ffffffff8125575a>] ? ep_poll+0x21a/0x3d0
  [ 2817.923714]  [<ffffffff8106b537>] __do_page_fault+0x197/0x400
  [ 2817.924503]  [<ffffffff8106b7c2>] do_page_fault+0x22/0x30
  [ 2817.925295]  [<ffffffff81827478>] page_fault+0x28/0x30

  [ 2818.096501] irqbalance      D ffff880153413b78     0  1005      1 0x00000000
  [ 2818.097266]  ffff880153413b78 ffff880094347080 ffff880159472940 ffff880095406040
  [ 2818.098039]  ffff880153414000 ffff88015ecd6d00 7fffffffffffffff ffffffff81821a00
  [ 2818.098814]  ffff880153413cd8 ffff880153413b90 ffffffff81821205 0000000000000000
  [ 2818.099591] Call Trace:
  [ 2818.100346]  [<ffffffff81821a00>] ? bit_wait+0x60/0x60
  [ 2818.101114]  [<ffffffff81821205>] schedule+0x35/0x80
  [ 2818.101871]  [<ffffffff81824325>] schedule_timeout+0x1b5/0x270
  [ 2818.102626]  [<ffffffff813b5cf3>] ? __blk_run_queue+0x33/0x40
  [ 2818.103376]  [<ffffffff813b5fba>] ? queue_unplugged+0x2a/0xb0
  [ 2818.104147]  [<ffffffff81821a00>] ? bit_wait+0x60/0x60
  [ 2818.104894]  [<ffffffff81820754>] io_schedule_timeout+0xa4/0x110
  [ 2818.105638]  [<ffffffff81821a1b>] bit_wait_io+0x1b/0x70
  [ 2818.106392]  [<ffffffff818215ad>] __wait_on_bit+0x5d/0x90
  [ 2818.107142]  [<ffffffff8118f79d>] wait_on_page_bit_killable+0xcd/0xf0
  [ 2818.107919]  [<ffffffff810c3ab0>] ? autoremove_wake_function+0x40/0x40
  [ 2818.108674]  [<ffffffff8118f844>] __lock_page_or_retry+0x84/0xa0
  [ 2818.109435]  [<ffffffff8118fa4d>] filemap_fault+0x1ed/0x3f0
  [ 2818.110196]  [<ffffffff811bc1f0>] __do_fault+0x50/0xe0
  [ 2818.110956]  [<ffffffff811bfc2b>] handle_mm_fault+0xf8b/0x1820
  [ 2818.111729]  [<ffffffff8106b537>] __do_page_fault+0x197/0x400
  [ 2818.112489]  [<ffffffff8106b7c2>] do_page_fault+0x22/0x30
  [ 2818.113254]  [<ffffffff81827478>] page_fault+0x28/0x30

  9. avahi-daemon is blocked in pagefault handler with different
  stacktrace

  [ 2818.403217] avahi-daemon    D ffff880090e671f8     0  3601      1 0x00000000
  [ 2818.404027]  ffff880090e671f8 ffffffff811eaac6 ffff8800350ce040 ffff880094910000
  [ 2818.404832]  ffff880090e68000 ffff880090e67230 ffff88015eccdd00 ffff88015eccdd00
  [ 2818.405629]  ffff880094910000 ffff880090e67210 ffffffff81821205 0000000100099ee5
  [ 2818.406422] Call Trace:
  [ 2818.407195]  [<ffffffff811eaac6>] ? ___slab_alloc+0x2a6/0x460
  [ 2818.407992]  [<ffffffff81821205>] schedule+0x35/0x80
  [ 2818.408775]  [<ffffffff81824299>] schedule_timeout+0x129/0x270
  [ 2818.409558]  [<ffffffff810ec370>] ? trace_event_raw_event_tick_stop+0x120/0x120
  [ 2818.410349]  [<ffffffff810f574c>] ? ktime_get+0x3c/0xb0
  [ 2818.411143]  [<ffffffff81820754>] io_schedule_timeout+0xa4/0x110
  [ 2818.411957]  [<ffffffff811900c8>] mempool_alloc+0x148/0x170
  [ 2818.412750]  [<ffffffff810c3a70>] ? wake_atomic_t_function+0x60/0x60
  [ 2818.413549]  [<ffffffff813b20ad>] bio_alloc_bioset+0xbd/0x260
  [ 2818.414342]  [<ffffffff816a33cf>] __split_and_process_bio+0x1ef/0x3e0
  [ 2818.415137]  [<ffffffff816a3629>] dm_make_request+0x69/0xc0
  [ 2818.415937]  [<ffffffff813ba502>] generic_make_request+0xf2/0x1d0
  [ 2818.416724]  [<ffffffff813ba656>] submit_bio+0x76/0x170
  [ 2818.417507]  [<ffffffff811d2548>] __swap_writepage+0x228/0x270
  [ 2818.418288]  [<ffffffff811d7bcf>] ? __frontswap_store+0xdf/0x110
  [ 2818.419067]  [<ffffffff811d25c9>] swap_writepage+0x39/0x70
  [ 2818.419857]  [<ffffffff811a0b6d>] pageout.isra.44+0x16d/0x280
  [ 2818.420639]  [<ffffffff811a31ca>] shrink_page_list+0x3ca/0x7a0
  [ 2818.421428]  [<ffffffff811a3c39>] shrink_inactive_list+0x209/0x520
  [ 2818.422212]  [<ffffffff811a48d3>] shrink_lruvec+0x593/0x750
  [ 2818.422997]  [<ffffffff811a4b7f>] shrink_zone+0xef/0x2e0
  [ 2818.423779]  [<ffffffff811a4edc>] do_try_to_free_pages+0x16c/0x3f0
  [ 2818.424558]  [<ffffffff811a522e>] try_to_free_pages+0xce/0x180
  [ 2818.425327]  [<ffffffff81196eda>] __alloc_pages_nodemask+0x64a/0xb60
  [ 2818.426100]  [<ffffffff811e091c>] alloc_pages_current+0x8c/0x110
  [ 2818.426869]  [<ffffffff8118d77b>] __page_cache_alloc+0xab/0xc0
  [ 2818.427652]  [<ffffffff8119bf8b>] __do_page_cache_readahead+0xeb/0x230
  [ 2818.428415]  [<ffffffff8118d08e>] ? find_get_entry+0x1e/0xa0
  [ 2818.429172]  [<ffffffff8118fbd5>] filemap_fault+0x375/0x3f0
  [ 2818.429932]  [<ffffffff811bc1f0>] __do_fault+0x50/0xe0
  [ 2818.430693]  [<ffffffff811bfc2b>] handle_mm_fault+0xf8b/0x1820
  [ 2818.431461]  [<ffffffff81220f60>] ? poll_select_copy_remaining+0x140/0x140
  [ 2818.432246]  [<ffffffff8106b537>] __do_page_fault+0x197/0x400
  [ 2818.433015]  [<ffffffff8106b7c2>] do_page_fault+0x22/0x30
  [ 2818.433782]  [<ffffffff81827478>] page_fault+0x28/0x30

  10. Memory hungry compilation job is blocked in pagefault handler with
  similar trace to avahi

  [ 2818.535321] cc1plus         D ffff880090bcb388     0  3977   3976 0x00000000
  [ 2818.536087]  ffff880090bcb388 ffffffff811eaac6 ffffffff81e11500 ffff880094915280
  [ 2818.536853]  ffff880090bcc000 ffff880090bcb3c0 ffff88015ec0dd00 ffff88015ec0dd00
  [ 2818.537618]  ffff880094915280 ffff880090bcb3a0 ffffffff81821205 0000000100099ee5
  [ 2818.538375] Call Trace:
  [ 2818.539128]  [<ffffffff811eaac6>] ? ___slab_alloc+0x2a6/0x460
  [ 2818.539897]  [<ffffffff81821205>] schedule+0x35/0x80
  [ 2818.540665]  [<ffffffff81824299>] schedule_timeout+0x129/0x270
  [ 2818.541431]  [<ffffffff810ec370>] ? trace_event_raw_event_tick_stop+0x120/0x120
  [ 2818.542201]  [<ffffffff810f574c>] ? ktime_get+0x3c/0xb0
  [ 2818.542970]  [<ffffffff81820754>] io_schedule_timeout+0xa4/0x110
  [ 2818.543748]  [<ffffffff811900c8>] mempool_alloc+0x148/0x170
  [ 2818.544517]  [<ffffffff810c3a70>] ? wake_atomic_t_function+0x60/0x60
  [ 2818.545286]  [<ffffffff813b20ad>] bio_alloc_bioset+0xbd/0x260
  [ 2818.546063]  [<ffffffff816a33cf>] __split_and_process_bio+0x1ef/0x3e0
  [ 2818.546840]  [<ffffffff816a3629>] dm_make_request+0x69/0xc0
  [ 2818.547632]  [<ffffffff813ba502>] generic_make_request+0xf2/0x1d0
  [ 2818.548416]  [<ffffffff813ba656>] submit_bio+0x76/0x170
  [ 2818.549200]  [<ffffffff811d2548>] __swap_writepage+0x228/0x270
  [ 2818.549980]  [<ffffffff811d7bcf>] ? __frontswap_store+0xdf/0x110
  [ 2818.550763]  [<ffffffff811d25c9>] swap_writepage+0x39/0x70
  [ 2818.551546]  [<ffffffff811a0b6d>] pageout.isra.44+0x16d/0x280
  [ 2818.552341]  [<ffffffff811a31ca>] shrink_page_list+0x3ca/0x7a0
  [ 2818.553121]  [<ffffffff811a3c39>] shrink_inactive_list+0x209/0x520
  [ 2818.553902]  [<ffffffff811a48d3>] shrink_lruvec+0x593/0x750
  [ 2818.554677]  [<ffffffff810986d1>] ? queue_work_on+0x31/0x40
  [ 2818.555460]  [<ffffffff810986d1>] ? queue_work_on+0x31/0x40
  [ 2818.556230]  [<ffffffff811a4b7f>] shrink_zone+0xef/0x2e0
  [ 2818.557004]  [<ffffffff811a4edc>] do_try_to_free_pages+0x16c/0x3f0
  [ 2818.557784]  [<ffffffff811a522e>] try_to_free_pages+0xce/0x180
  [ 2818.558561]  [<ffffffff81196eda>] __alloc_pages_nodemask+0x64a/0xb60
  [ 2818.559337]  [<ffffffff811e216e>] alloc_pages_vma+0xbe/0x240
  [ 2818.560123]  [<ffffffff811c012e>] handle_mm_fault+0x148e/0x1820
  [ 2818.560884]  [<ffffffff8106b537>] __do_page_fault+0x197/0x400
  [ 2818.561643]  [<ffffffff8106b7c2>] do_page_fault+0x22/0x30
  [ 2818.562397]  [<ffffffff81827478>] page_fault+0x28/0x30

  11. disk events work is blocked in lru shrink

  [ 2818.563150] kworker/u16:5   D ffff8800958135c8     0  3978      2 0x00000000
  [ 2818.563922] Workqueue: events_freezable_power_ disk_events_workfn
  [ 2818.564681]  ffff8800958135c8 ffffffff811cc3a6 ffffffff81e11500 ffff8800350ce040
  [ 2818.565456]  ffff880095814000 ffff880095813600 ffff88015ec0dd00 ffff88015ec0dd00
  [ 2818.566234]  ffffffffffffffe0 ffff8800958135e0 ffffffff81821205 0000000100099b6b
  [ 2818.567022] Call Trace:
  [ 2818.567803]  [<ffffffff811cc3a6>] ? page_referenced+0xb6/0x140
  [ 2818.568593]  [<ffffffff81821205>] schedule+0x35/0x80
  [ 2818.569372]  [<ffffffff81824299>] schedule_timeout+0x129/0x270
  [ 2818.570152]  [<ffffffff810ec370>] ? trace_event_raw_event_tick_stop+0x120/0x120
  [ 2818.570936]  [<ffffffff8182443e>] schedule_timeout_uninterruptible+0x1e/0x20
  [ 2818.571720]  [<ffffffff811aea13>] wait_iff_congested+0xf3/0x180
  [ 2818.572505]  [<ffffffff810c3a70>] ? wake_atomic_t_function+0x60/0x60
  [ 2818.573283]  [<ffffffff811a3f2f>] shrink_inactive_list+0x4ff/0x520
  [ 2818.574056]  [<ffffffff811a48d3>] shrink_lruvec+0x593/0x750
  [ 2818.574829]  [<ffffffff8120ebc0>] ? super_cache_count+0x70/0xe0
  [ 2818.575610]  [<ffffffff811a4b7f>] shrink_zone+0xef/0x2e0
  [ 2818.576381]  [<ffffffff811a4edc>] do_try_to_free_pages+0x16c/0x3f0
  [ 2818.577158]  [<ffffffff811a522e>] try_to_free_pages+0xce/0x180
  [ 2818.577941]  [<ffffffff81196eda>] __alloc_pages_nodemask+0x64a/0xb60
  [ 2818.578717]  [<ffffffff813b60b9>] ? alloc_request_struct+0x19/0x20
  [ 2818.579488]  [<ffffffff811e091c>] alloc_pages_current+0x8c/0x110
  [ 2818.580262]  [<ffffffff813b33e7>] bio_copy_kern+0xd7/0x1e0
  [ 2818.581030]  [<ffffffff813c0088>] blk_rq_map_kern+0xa8/0x140
  [ 2818.581797]  [<ffffffff813b9e5b>] ? blk_get_request+0x8b/0xf0
  [ 2818.582558]  [<ffffffff815adc68>] scsi_execute+0x78/0x1d0
  [ 2818.583317]  [<ffffffff815af96e>] scsi_execute_req_flags+0x8e/0xf0
  [ 2818.584087]  [<ffffffff815c0e27>] sr_check_events+0xb7/0x2d0
  [ 2818.584845]  [<ffffffff815fe0d8>] cdrom_check_events+0x18/0x30
  [ 2818.585610]  [<ffffffff815c125a>] sr_block_check_events+0x2a/0x30
  [ 2818.586367]  [<ffffffff813cb5f0>] disk_check_events+0x60/0x150
  [ 2818.587119]  [<ffffffff813cb6f6>] disk_events_workfn+0x16/0x20
  [ 2818.587875]  [<ffffffff8109a052>] process_one_work+0x162/0x480
  [ 2818.588624]  [<ffffffff8109a3bb>] worker_thread+0x4b/0x4c0
  [ 2818.589353]  [<ffffffff8109a370>] ? process_one_work+0x480/0x480
  [ 2818.590090]  [<ffffffff8109a370>] ? process_one_work+0x480/0x480
  [ 2818.590817]  [<ffffffff810a0588>] kthread+0xd8/0xf0
  [ 2818.591535]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
  [ 2818.592274]  [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
  [ 2818.592993]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0

  
  12. kernel workqueues have a lot of stuf waiting execution

  [ 2819.710760]
  [ 2819.711509] Showing busy workqueues and worker pools:
  [ 2819.712278] workqueue events_freezable_power_: flags=0x86
  [ 2819.713053]   pwq 16: cpus=0-7 flags=0x4 nice=0 active=1/256
  [ 2819.713843]     in-flight: 3978:disk_events_workfn
  [ 2819.714673] workqueue kcryptd: flags=0x2a
  [ 2819.715464]   pwq 16: cpus=0-7 flags=0x4 nice=0 active=4/4
  [ 2819.716267]     in-flight: 3966:kcryptd_crypt [dm_crypt], 160:kcryptd_crypt [dm_crypt], 3388:kcryptd_crypt [dm_crypt], 3339:kcryptd_crypt [dm_crypt]
  [ 2819.717131]     delayed: kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], <repeats enought times to make line 17983 characters wide>
  [ 2822.339640] pool 16: cpus=0-7 flags=0x4 nice=0 workers=7 idle: 3979 3139

  13. Swap usage stats from kernel:
  [ 2960.187650] Swap cache stats: add 16096, delete 3484, find 193/223
  [ 2960.187711] Free swap  = 4054404kB
  [ 2960.187748] Total swap = 4112380kB


  Recovery from hang state:
  Sometimes hang state can be recovred if I use sysrq-k and sysrq-e to kill all userspace programs. Systemd will then restart userspace process if hanged state was recovered successfully. Recovering doesn't work 

  
  Workaround:
  Disabling swap with swapoff -a.
  Running same compile job results to OOM killer cleanly killing the memory hungry process.

  ProblemType: Bug
  DistroRelease: Ubuntu 16.04
  Package: linux-image-4.4.0-22-generic 4.4.0-22.40
  ProcVersionSignature: Ubuntu 4.4.0-22.40-generic 4.4.8
  Uname: Linux 4.4.0-22-generic x86_64
  NonfreeKernelModules: nvidia
  ApportVersion: 2.20.1-0ubuntu2.1
  Architecture: amd64
  AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path', '/dev/snd/hwC1D0', '/dev/snd/pcmC1D0c', '/dev/snd/pcmC1D0p', '/dev/snd/controlC1', '/dev/snd/hwC0D0', '/dev/snd/pcmC0D8p', '/dev/snd/pcmC0D7p', '/dev/snd/pcmC0D3p', '/dev/snd/controlC0', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
  CurrentDesktop: GNOME
  Date: Mon Jun  6 17:55:26 2016
  HibernationDevice: RESUME=UUID=04553608-429c-40dc-b53c-a200f2a70588
  InstallationDate: Installed on 2037-12-25 (-7871 days ago)
  InstallationMedia: Lubuntu 16.04 LTS "Xenial Xerus" - Release amd64 (20160420.1)
  MachineType: Acer Aspire E5-571G
  ProcFB: 0 inteldrmfb
  ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-22-generic root=/dev/mapper/lubuntu--vg-root ro quiet splash
  PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
  RelatedPackageVersions:
   linux-restricted-modules-4.4.0-22-generic N/A
   linux-backports-modules-4.4.0-22-generic  N/A
   linux-firmware                            1.157
  SourcePackage: linux
  UpgradeStatus: No upgrade log present (probably fresh install)
  dmi.bios.date: 09/15/2015
  dmi.bios.vendor: Insyde Corp.
  dmi.bios.version: V1.32
  dmi.board.name: EA50_HB
  dmi.board.vendor: Acer
  dmi.board.version: V1.32
  dmi.chassis.type: 10
  dmi.chassis.vendor: Acer
  dmi.chassis.version: V1.32
  dmi.modalias: dmi:bvnInsydeCorp.:bvrV1.32:bd09/15/2015:svnAcer:pnAspireE5-571G:pvrV1.32:rvnAcer:rnEA50_HB:rvrV1.32:cvnAcer:ct10:cvrV1.32:
  dmi.product.name: Aspire E5-571G
  dmi.product.version: V1.32
  dmi.sys.vendor: Acer

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1589563/+subscriptions


References