kernel-packages team mailing list archive
-
kernel-packages team
-
Mailing list archive
-
Message #182495
[Bug 1589563] [NEW] ext4 journaling and swapping to same encrypted SSHD hangs system
Public bug reported:
Short description:
The system has single SSHD disk with luks/lvm2 encrypted root file system and swap partition. If system starts to swap ext4 journal commit and swap requests hang waiting disk io.
Relevant bug reports in other places:
So far I have managed to find only one relevant looking report about very similar problem
https://bbs.archlinux.org/viewtopic.php?id=193034
Hardware:
Acer Aspire E15 E5-571G59EQ
Intel Broadwell i5-5200U
4GB ram
nvidia GeForce 840M (binary driver installed but disabled)
Seagate ST500LM000-1EJ16 500GB HDD + 8GB SSHD
Known ways to reproduce:
Compiling templated c++ code using g++ 5 that results to g++ optimization taking huge amount of memory.
I haven't yet tested if there is any other
Reproduction details:
$ cat main.cpp
#include <iostream>
template<unsigned idx>
unsigned loop()
{
if (idx == 1)
return 6;
unsigned count = 0;
for (unsigned i = 1; i < 7; i++) {
count += loop< (idx > 1 ? idx - 1 : 1) >();
}
return count;
}
int main(void)
{
unsigned count, dice[5];
std::cout << "6^5: " << 6*6*6*6*6 << "\n";
std::cout << "loop all: " << loop<5>() << "\n";
std::cout << "loop all two players: " << loop<10>() << "\n";
return 0;
}
$ gcc --version
gcc (Ubuntu 5.3.1-14ubuntu2.1) 5.3.1 20160413
$ g++ -O2 -g -c -o main.o main.cpp
Relevant backtraces from sysrq-t (recorded using netconsole):
1. Watchdog catches jdb2 hang
[ 2784.570413] INFO: task jbd2/dm-1-8:372 blocked for more than 10 seconds.
[ 2784.571209] Tainted: P OE 4.4.0-22-generic #40-Ubuntu
[ 2784.571948] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2784.572606] jbd2/dm-1-8 D ffff880035577ad8 0 372 2 0x00000000
[ 2784.573294] ffff880035577ad8 ffff880035577b00 ffff880159472940 ffff8801561e0dc0
[ 2784.573997] ffff880035578000 ffff88015ecd6d00 7fffffffffffffff ffffffff81821a00
[ 2784.574724] ffff880035577c30 ffff880035577af0 ffffffff81821205 0000000000000000
[ 2784.575445] Call Trace:
[ 2784.576124] [<ffffffff81821a00>] ? bit_wait+0x60/0x60
[ 2784.576835] [<ffffffff81821205>] schedule+0x35/0x80
[ 2784.577546] [<ffffffff81824325>] schedule_timeout+0x1b5/0x270
[ 2784.578255] [<ffffffff813b5cf3>] ? __blk_run_queue+0x33/0x40
[ 2784.578959] [<ffffffff813bc3d0>] ? blk_queue_bio+0x3d0/0x3e0
[ 2784.579635] [<ffffffff810f574c>] ? ktime_get+0x3c/0xb0
[ 2784.580328] [<ffffffff81821a00>] ? bit_wait+0x60/0x60
[ 2784.581047] [<ffffffff81820754>] io_schedule_timeout+0xa4/0x110
[ 2784.581767] [<ffffffff81821a1b>] bit_wait_io+0x1b/0x70
[ 2784.582509] [<ffffffff818215ad>] __wait_on_bit+0x5d/0x90
[ 2784.583215] [<ffffffff81821a00>] ? bit_wait+0x60/0x60
[ 2784.583951] [<ffffffff81821662>] out_of_line_wait_on_bit+0x82/0xb0
[ 2784.584687] [<ffffffff810c3ab0>] ? autoremove_wake_function+0x40/0x40
[ 2784.585434] [<ffffffff81243c32>] __wait_on_buffer+0x32/0x40
[ 2784.586174] [<ffffffff812eaf6f>] jbd2_journal_commit_transaction+0x10cf/0x1870
[ 2784.586906] [<ffffffff810ec5fe>] ? try_to_del_timer_sync+0x5e/0x90
[ 2784.587622] [<ffffffff812ef32a>] kjournald2+0xca/0x250
[ 2784.588339] [<ffffffff810c3a70>] ? wake_atomic_t_function+0x60/0x60
[ 2784.589058] [<ffffffff812ef260>] ? commit_timeout+0x10/0x10
[ 2784.589769] [<ffffffff810a0588>] kthread+0xd8/0xf0
[ 2784.590503] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2784.591215] [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[ 2784.591930] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
2. kswaped looks like hanged too
[ 2817.029410] kswapd0 D ffff8800934ab608 0 43 2 0x00000000
[ 2817.030212] ffff8800934ab608 ffffffff811eaac6 ffff880093b6b700 ffff8800934a0000
[ 2817.031032] ffff8800934ac000 ffff8800934ab640 ffff88015eccdd00 ffff88015eccdd00
[ 2817.031860] ffff8800934a0000 ffff8800934ab620 ffffffff81821205 0000000100099a03
[ 2817.032670] Call Trace:
[ 2817.033473] [<ffffffff811eaac6>] ? ___slab_alloc+0x2a6/0x460
[ 2817.034282] [<ffffffff81821205>] schedule+0x35/0x80
[ 2817.035082] [<ffffffff81824299>] schedule_timeout+0x129/0x270
[ 2817.035898] [<ffffffff810ec370>] ? trace_event_raw_event_tick_stop+0x120/0x120
[ 2817.036709] [<ffffffff810f574c>] ? ktime_get+0x3c/0xb0
[ 2817.037515] [<ffffffff81820754>] io_schedule_timeout+0xa4/0x110
[ 2817.038327] [<ffffffff811900c8>] mempool_alloc+0x148/0x170
[ 2817.039137] [<ffffffff810c3a70>] ? wake_atomic_t_function+0x60/0x60
[ 2817.039964] [<ffffffff813b20ad>] bio_alloc_bioset+0xbd/0x260
[ 2817.040774] [<ffffffff816a33cf>] __split_and_process_bio+0x1ef/0x3e0
[ 2817.041576] [<ffffffff816a3629>] dm_make_request+0x69/0xc0
[ 2817.042377] [<ffffffff813ba502>] generic_make_request+0xf2/0x1d0
[ 2817.043180] [<ffffffff813ba656>] submit_bio+0x76/0x170
[ 2817.043999] [<ffffffff811d2548>] __swap_writepage+0x228/0x270
[ 2817.044799] [<ffffffff811d7bcf>] ? __frontswap_store+0xdf/0x110
[ 2817.045589] [<ffffffff811d25c9>] swap_writepage+0x39/0x70
[ 2817.046386] [<ffffffff811a0b6d>] pageout.isra.44+0x16d/0x280
[ 2817.047179] [<ffffffff811a31ca>] shrink_page_list+0x3ca/0x7a0
[ 2817.047990] [<ffffffff811a3c39>] shrink_inactive_list+0x209/0x520
[ 2817.048785] [<ffffffff811a48d3>] shrink_lruvec+0x593/0x750
[ 2817.049582] [<ffffffff81098239>] ? __queue_work+0x139/0x3b0
[ 2817.050373] [<ffffffff811a4b7f>] shrink_zone+0xef/0x2e0
[ 2817.051159] [<ffffffff811a5dfe>] kswapd+0x51e/0x990
[ 2817.051952] [<ffffffff811a58e0>] ? mem_cgroup_shrink_node_zone+0x1c0/0x1c0
[ 2817.052731] [<ffffffff810a0588>] kthread+0xd8/0xf0
[ 2817.053512] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2817.054299] [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[ 2817.055083] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
3. encyptfs-thread and bioset looks like sleeping without work
[ 2817.076543] ecryptfs-kthrea S ffff880093553e40 0 46 2 0x00000000
[ 2817.077347] ffff880093553e40 00000000934a29a0 ffff880159472940 ffff8800934a2940
[ 2817.078160] ffff880093554000 ffff880093553e80 ffff8800934a2940 0000000000000000
[ 2817.078980] 0000000000000000 ffff880093553e58 ffffffff81821205 0000000000000000
[ 2817.079795] Call Trace:
[ 2817.080596] [<ffffffff81821205>] schedule+0x35/0x80
[ 2817.081407] [<ffffffff8130b02a>] ecryptfs_threadfn+0x17a/0x1c0
[ 2817.082212] [<ffffffff810c3a70>] ? wake_atomic_t_function+0x60/0x60
[ 2817.083017] [<ffffffff8130aeb0>] ? ecryptfs_add_global_auth_tok+0xa0/0xa0
[ 2817.083829] [<ffffffff810a0588>] kthread+0xd8/0xf0
[ 2817.084628] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2817.085432] [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[ 2817.086227] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
4. scsi threads seem to be sleeping (for all cores same)
[ 2817.390470] scsi_eh_0 S ffff8800357c7e10 0 152 2 0x00000000
[ 2817.391287] ffff8800357c7e10 ffff8800357c7e08 ffff880159472940 ffff8800937a5280
[ 2817.392107] ffff8800357c8000 ffff8800937a5858 ffff8800937a5858 ffff8800937a5280
[ 2817.392917] 0000000000000000 ffff8800357c7e28 ffffffff81821205 ffff8800356c9800
[ 2817.393730] Call Trace:
[ 2817.394537] [<ffffffff81821205>] schedule+0x35/0x80
[ 2817.395339] [<ffffffff815ad2f7>] scsi_error_handler+0x97/0x8a0
[ 2817.396142] [<ffffffff81820b46>] ? __schedule+0x386/0xa10
[ 2817.396946] [<ffffffff815ad260>] ? scsi_eh_get_sense+0x250/0x250
[ 2817.397748] [<ffffffff810a0588>] kthread+0xd8/0xf0
[ 2817.398553] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2817.399363] [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[ 2817.400170] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2817.400977] scsi_tmf_0 S ffff8800357cbe38 0 153 2 0x00000000
[ 2817.401800] ffff8800357cbe38 ffff8800357cbe40 ffff880159470dc0 ffff8800937a44c0
[ 2817.402624] ffff8800357cc000 ffff8800937a44c0 ffffffff8109a830 ffff8800937e9a80
[ 2817.403439] ffff8800356e0000 ffff8800357cbe50 ffffffff81821205 ffff8800356e0030
[ 2817.404257] Call Trace:
[ 2817.405064] [<ffffffff8109a830>] ? worker_thread+0x4c0/0x4c0
[ 2817.405870] [<ffffffff81821205>] schedule+0x35/0x80
[ 2817.406673] [<ffffffff8109ab75>] rescuer_thread+0x345/0x3d0
[ 2817.407475] [<ffffffff8109a830>] ? worker_thread+0x4c0/0x4c0
[ 2817.408279] [<ffffffff810a0588>] kthread+0xd8/0xf0
[ 2817.409083] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2817.409889] [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[ 2817.410699] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
5. kworker running dm_crypt work seem to be waiting for something in lru
shrink
[ 2817.474917] kworker/u16:3 R running task 0 160 2 0x00000000
[ 2817.475748] Workqueue: kcryptd kcryptd_crypt [dm_crypt]
[ 2817.476566] ffff880035007538 ffffffff811cc3a6 ffff88015723b700 ffff8800354044c0
[ 2817.477396] ffff880035008000 ffff880035007570 ffff88015ec0dd00 ffff88015ec0dd00
[ 2817.478218] ffffffffffffffe0 ffff880035007550 ffffffff81821205 0000000100099990
[ 2817.479052] Call Trace:
[ 2817.479866] [<ffffffff811cc3a6>] ? page_referenced+0xb6/0x140
[ 2817.480694] [<ffffffff81821205>] schedule+0x35/0x80
[ 2817.481520] [<ffffffff81824299>] schedule_timeout+0x129/0x270
[ 2817.482337] [<ffffffff810ec370>] ? trace_event_raw_event_tick_stop+0x120/0x120
[ 2817.483173] [<ffffffff8182443e>] schedule_timeout_uninterruptible+0x1e/0x20
[ 2817.484009] [<ffffffff811aea13>] wait_iff_congested+0xf3/0x180
[ 2817.484847] [<ffffffff810c3a70>] ? wake_atomic_t_function+0x60/0x60
[ 2817.485682] [<ffffffff811a3f2f>] shrink_inactive_list+0x4ff/0x520
[ 2817.486513] [<ffffffff811a48d3>] shrink_lruvec+0x593/0x750
[ 2817.487332] [<ffffffff811a4b7f>] shrink_zone+0xef/0x2e0
[ 2817.488151] [<ffffffff811a4edc>] do_try_to_free_pages+0x16c/0x3f0
[ 2817.488978] [<ffffffff811a522e>] try_to_free_pages+0xce/0x180
[ 2817.489795] [<ffffffff81196eda>] __alloc_pages_nodemask+0x64a/0xb60
[ 2817.490608] [<ffffffff811ade4c>] ? zone_statistics+0x7c/0xa0
[ 2817.491423] [<ffffffff811e091c>] alloc_pages_current+0x8c/0x110
[ 2817.492237] [<ffffffff811e99ff>] new_slab+0x28f/0x490
[ 2817.493050] [<ffffffff811eaa4b>] ___slab_alloc+0x22b/0x460
[ 2817.493864] [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
[ 2817.494663] [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
[ 2817.495466] [<ffffffff811eaca0>] __slab_alloc+0x20/0x40
[ 2817.496264] [<ffffffff811eb21f>] kmem_cache_alloc+0x19f/0x1f0
[ 2817.497065] [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
[ 2817.497858] [<ffffffff8118ff05>] mempool_alloc_slab+0x15/0x20
[ 2817.498644] [<ffffffff8118ffee>] mempool_alloc+0x6e/0x170
[ 2817.499429] [<ffffffffc0162092>] ? __ablk_encrypt+0x52/0x70 [ablk_helper]
[ 2817.500218] [<ffffffff813b20ad>] bio_alloc_bioset+0xbd/0x260
[ 2817.501003] [<ffffffffc0112af4>] kcryptd_crypt+0x104/0x380 [dm_crypt]
[ 2817.501787] [<ffffffff8109a052>] process_one_work+0x162/0x480
[ 2817.502572] [<ffffffff8109a3bb>] worker_thread+0x4b/0x4c0
[ 2817.503356] [<ffffffff8109a370>] ? process_one_work+0x480/0x480
[ 2817.504135] [<ffffffff810a0588>] kthread+0xd8/0xf0
[ 2817.504919] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2817.505704] [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[ 2817.506490] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2818.290300] kworker/u16:1 D ffff8801561fb538 0 3339 2 0x00000000
[ 2818.291008] Workqueue: kcryptd kcryptd_crypt [dm_crypt]
[ 2818.291734] ffff8801561fb538 ffffffff811cc3a6 ffff880159472940 ffff880093b82940
[ 2818.292466] ffff8801561fc000 ffff8801561fb570 ffff88015eccdd00 ffff88015eccdd00
[ 2818.293201] ffffffffffffffe0 ffff8801561fb550 ffffffff81821205 0000000100099b27
[ 2818.293938] Call Trace:
[ 2818.294658] [<ffffffff811cc3a6>] ? page_referenced+0xb6/0x140
[ 2818.295400] [<ffffffff81821205>] schedule+0x35/0x80
[ 2818.296162] [<ffffffff81824299>] schedule_timeout+0x129/0x270
[ 2818.296924] [<ffffffff810ec370>] ? trace_event_raw_event_tick_stop+0x120/0x120
[ 2818.297700] [<ffffffff8182443e>] schedule_timeout_uninterruptible+0x1e/0x20
[ 2818.298479] [<ffffffff811aea13>] wait_iff_congested+0xf3/0x180
[ 2818.299264] [<ffffffff810c3a70>] ? wake_atomic_t_function+0x60/0x60
[ 2818.300063] [<ffffffff811a3f2f>] shrink_inactive_list+0x4ff/0x520
[ 2818.300853] [<ffffffff811a48d3>] shrink_lruvec+0x593/0x750
[ 2818.301645] [<ffffffff811a4b7f>] shrink_zone+0xef/0x2e0
[ 2818.302437] [<ffffffff811a4edc>] do_try_to_free_pages+0x16c/0x3f0
[ 2818.303232] [<ffffffff811a522e>] try_to_free_pages+0xce/0x180
[ 2818.304038] [<ffffffff81196eda>] __alloc_pages_nodemask+0x64a/0xb60
[ 2818.304831] [<ffffffff811ade4c>] ? zone_statistics+0x7c/0xa0
[ 2818.305619] [<ffffffff811e091c>] alloc_pages_current+0x8c/0x110
[ 2818.306399] [<ffffffff811e99ff>] new_slab+0x28f/0x490
[ 2818.307183] [<ffffffff811eaa4b>] ___slab_alloc+0x22b/0x460
[ 2818.307975] [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
[ 2818.308751] [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
[ 2818.309513] [<ffffffff811eaca0>] __slab_alloc+0x20/0x40
[ 2818.310279] [<ffffffff811eb21f>] kmem_cache_alloc+0x19f/0x1f0
[ 2818.311049] [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
[ 2818.311834] [<ffffffff8118ff05>] mempool_alloc_slab+0x15/0x20
[ 2818.312603] [<ffffffff8118ffee>] mempool_alloc+0x6e/0x170
[ 2818.313377] [<ffffffffc0162092>] ? __ablk_encrypt+0x52/0x70 [ablk_helper]
[ 2818.314150] [<ffffffff813b20ad>] bio_alloc_bioset+0xbd/0x260
[ 2818.314925] [<ffffffffc0112af4>] kcryptd_crypt+0x104/0x380 [dm_crypt]
[ 2818.315712] [<ffffffff8109a052>] process_one_work+0x162/0x480
[ 2818.316484] [<ffffffff8109a3bb>] worker_thread+0x4b/0x4c0
[ 2818.317250] [<ffffffff8109a370>] ? process_one_work+0x480/0x480
[ 2818.318018] [<ffffffff810a0588>] kthread+0xd8/0xf0
[ 2818.318782] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2818.319561] [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[ 2818.320333] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2818.321113] kworker/u16:2 D ffff880098653538 0 3388 2 0x00000000
[ 2818.321909] Workqueue: kcryptd kcryptd_crypt [dm_crypt]
[ 2818.322700] ffff880098653538 ffffffff811cc3a6 ffff880159471b80 ffff880093b6b700
[ 2818.323508] ffff880098654000 ffff880098653570 ffff88015ec8dd00 ffff88015ec8dd00
[ 2818.324313] ffffffffffffffe0 ffff880098653550 ffffffff81821205 0000000100099990
[ 2818.325116] Call Trace:
[ 2818.325905] [<ffffffff811cc3a6>] ? page_referenced+0xb6/0x140
[ 2818.326705] [<ffffffff81821205>] schedule+0x35/0x80
[ 2818.327498] [<ffffffff81824299>] schedule_timeout+0x129/0x270
[ 2818.328301] [<ffffffff810ec370>] ? trace_event_raw_event_tick_stop+0x120/0x120
[ 2818.329103] [<ffffffff8182443e>] schedule_timeout_uninterruptible+0x1e/0x20
[ 2818.329908] [<ffffffff811aea13>] wait_iff_congested+0xf3/0x180
[ 2818.330711] [<ffffffff810c3a70>] ? wake_atomic_t_function+0x60/0x60
[ 2818.331516] [<ffffffff811a3f2f>] shrink_inactive_list+0x4ff/0x520
[ 2818.332332] [<ffffffff811a48d3>] shrink_lruvec+0x593/0x750
[ 2818.333128] [<ffffffff811a4b7f>] shrink_zone+0xef/0x2e0
[ 2818.333912] [<ffffffff811a4edc>] do_try_to_free_pages+0x16c/0x3f0
[ 2818.334698] [<ffffffff811a522e>] try_to_free_pages+0xce/0x180
[ 2818.335483] [<ffffffff81194d44>] ? drain_local_pages+0x24/0x30
[ 2818.336278] [<ffffffff81196eda>] __alloc_pages_nodemask+0x64a/0xb60
[ 2818.337060] [<ffffffff811ade4c>] ? zone_statistics+0x7c/0xa0
[ 2818.337844] [<ffffffff811e091c>] alloc_pages_current+0x8c/0x110
[ 2818.338628] [<ffffffff811e99ff>] new_slab+0x28f/0x490
[ 2818.339407] [<ffffffff811eaa4b>] ___slab_alloc+0x22b/0x460
[ 2818.340204] [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
[ 2818.340982] [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
[ 2818.341756] [<ffffffff811eaca0>] __slab_alloc+0x20/0x40
[ 2818.342517] [<ffffffff811eb21f>] kmem_cache_alloc+0x19f/0x1f0
[ 2818.343276] [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
[ 2818.344044] [<ffffffff8118ff05>] mempool_alloc_slab+0x15/0x20
[ 2818.344793] [<ffffffff8118ffee>] mempool_alloc+0x6e/0x170
[ 2818.345540] [<ffffffffc0162092>] ? __ablk_encrypt+0x52/0x70 [ablk_helper]
[ 2818.346287] [<ffffffff813b20ad>] bio_alloc_bioset+0xbd/0x260
[ 2818.347022] [<ffffffffc0112af4>] kcryptd_crypt+0x104/0x380 [dm_crypt]
[ 2818.347775] [<ffffffff8109a052>] process_one_work+0x162/0x480
[ 2818.348518] [<ffffffff8109a3bb>] worker_thread+0x4b/0x4c0
[ 2818.349260] [<ffffffff8109a370>] ? process_one_work+0x480/0x480
[ 2818.349990] [<ffffffff810a0588>] kthread+0xd8/0xf0
[ 2818.350722] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2818.351456] [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[ 2818.352199] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
2818.478112] kworker/u16:4 D ffff880094c5b538 0 3966 2 0x00000000
[ 2818.478856] Workqueue: kcryptd kcryptd_crypt [dm_crypt]
[ 2818.479610] ffff880094c5b538 ffffffff811cc3a6 ffff880159470dc0 ffff88015723b700
[ 2818.480374] ffff880094c5c000 ffff880094c5b570 ffff88015ec4dd00 ffff88015ec4dd00
[ 2818.481137] ffffffffffffffe0 ffff880094c5b550 ffffffff81821205 0000000100099b56
[ 2818.481901] Call Trace:
[ 2818.482651] [<ffffffff811cc3a6>] ? page_referenced+0xb6/0x140
[ 2818.483413] [<ffffffff81821205>] schedule+0x35/0x80
[ 2818.484186] [<ffffffff81824299>] schedule_timeout+0x129/0x270
[ 2818.484940] [<ffffffff810ec370>] ? trace_event_raw_event_tick_stop+0x120/0x120
[ 2818.485691] [<ffffffff8182443e>] schedule_timeout_uninterruptible+0x1e/0x20
[ 2818.486447] [<ffffffff811aea13>] wait_iff_congested+0xf3/0x180
[ 2818.487201] [<ffffffff810c3a70>] ? wake_atomic_t_function+0x60/0x60
[ 2818.487964] [<ffffffff811a3f2f>] shrink_inactive_list+0x4ff/0x520
[ 2818.488712] [<ffffffff811a48d3>] shrink_lruvec+0x593/0x750
[ 2818.489468] [<ffffffff811a4b7f>] shrink_zone+0xef/0x2e0
[ 2818.490209] [<ffffffff811a4edc>] do_try_to_free_pages+0x16c/0x3f0
[ 2818.490951] [<ffffffff811a522e>] try_to_free_pages+0xce/0x180
[ 2818.491711] [<ffffffff81194d44>] ? drain_local_pages+0x24/0x30
[ 2818.492459] [<ffffffff81196eda>] __alloc_pages_nodemask+0x64a/0xb60
[ 2818.493202] [<ffffffff811ade4c>] ? zone_statistics+0x7c/0xa0
[ 2818.493944] [<ffffffff811e091c>] alloc_pages_current+0x8c/0x110
[ 2818.494696] [<ffffffff811e99ff>] new_slab+0x28f/0x490
[ 2818.495448] [<ffffffff811eaa4b>] ___slab_alloc+0x22b/0x460
[ 2818.496210] [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
[ 2818.496964] [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
[ 2818.497715] [<ffffffff811eaca0>] __slab_alloc+0x20/0x40
[ 2818.498469] [<ffffffff811eb21f>] kmem_cache_alloc+0x19f/0x1f0
[ 2818.499220] [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
[ 2818.499973] [<ffffffff8118ff05>] mempool_alloc_slab+0x15/0x20
[ 2818.500708] [<ffffffff8118ffee>] mempool_alloc+0x6e/0x170
[ 2818.501437] [<ffffffffc0162092>] ? __ablk_encrypt+0x52/0x70 [ablk_helper]
[ 2818.502169] [<ffffffff813b20ad>] bio_alloc_bioset+0xbd/0x260
[ 2818.502888] [<ffffffffc0112af4>] kcryptd_crypt+0x104/0x380 [dm_crypt]
[ 2818.503624] [<ffffffff8109a052>] process_one_work+0x162/0x480
[ 2818.504346] [<ffffffff8109a3bb>] worker_thread+0x4b/0x4c0
[ 2818.505069] [<ffffffff8109a370>] ? process_one_work+0x480/0x480
[ 2818.505794] [<ffffffff8109a370>] ? process_one_work+0x480/0x480
[ 2818.506496] [<ffffffff810a0588>] kthread+0xd8/0xf0
[ 2818.507205] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2818.507930] [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[ 2818.508642] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
6. kcryptd seems to be sleeping without work
[ 2817.579092] kcryptd_io S ffff8800352ebe38 0 318 2 0x00000000
[ 2817.579901] ffff8800352ebe38 ffff8800352ebe40 ffffffff81e11500 ffff880035068dc0
[ 2817.580712] ffff8800352ec000 ffff880035068dc0 ffffffff8109a830 ffff880035686a80
[ 2817.581516] ffff8800937d9180 ffff8800352ebe50 ffffffff81821205 ffff8800937d91b0
[ 2817.582314] Call Trace:
[ 2817.583104] [<ffffffff8109a830>] ? worker_thread+0x4c0/0x4c0
[ 2817.583904] [<ffffffff81821205>] schedule+0x35/0x80
[ 2817.584699] [<ffffffff8109ab75>] rescuer_thread+0x345/0x3d0
[ 2817.585494] [<ffffffff8109a830>] ? worker_thread+0x4c0/0x4c0
[ 2817.586284] [<ffffffff810a0588>] kthread+0xd8/0xf0
[ 2817.587076] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2817.587870] [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[ 2817.588663] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2817.589463] kcryptd S ffff8800352efe38 0 319 2 0x00000000
[ 2817.590270] ffff8800352efe38 ffff8800352efe40 ffffffff81e11500 ffff880035069b80
[ 2817.591078] ffff8800352f0000 ffff880035069b80 ffffffff8109a830 ffff8800355b9480
[ 2817.591881] ffff880156062c00 ffff8800352efe50 ffffffff81821205 ffff880156062c30
[ 2817.592681] Call Trace:
[ 2817.593468] [<ffffffff8109a830>] ? worker_thread+0x4c0/0x4c0
[ 2817.594250] [<ffffffff81821205>] schedule+0x35/0x80
[ 2817.595034] [<ffffffff8109ab75>] rescuer_thread+0x345/0x3d0
[ 2817.595828] [<ffffffff8109a830>] ? worker_thread+0x4c0/0x4c0
[ 2817.596618] [<ffffffff810a0588>] kthread+0xd8/0xf0
[ 2817.597413] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2817.598214] [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[ 2817.599014] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
7. dmcrypt_write seems like sleeping without work
[ 2817.599818] dmcrypt_write S ffff8800350f3de8 0 320 2 0x00000000
[ 2817.600626] ffff8800350f3de8 ffff8800350f3da8 ffff880159470dc0 ffff88003506a940
[ 2817.601435] ffff8800350f4000 ffff8800350f3e40 ffff8800355b9670 ffff88003506a940
[ 2817.602237] ffff8800355b9600 ffff8800350f3e00 ffffffff81821205 ffff8800355b9668
[ 2817.603038] Call Trace:
[ 2817.603827] [<ffffffff81821205>] schedule+0x35/0x80
[ 2817.604618] [<ffffffffc0113081>] dmcrypt_write+0xd1/0x1e0 [dm_crypt]
[ 2817.605412] [<ffffffff810abe40>] ? wake_up_q+0x70/0x70
[ 2817.606204] [<ffffffffc0112fb0>] ? crypt_iv_lmk_gen+0xc0/0xc0 [dm_crypt]
[ 2817.607008] [<ffffffffc0112fb0>] ? crypt_iv_lmk_gen+0xc0/0xc0 [dm_crypt]
[ 2817.607807] [<ffffffff810a0588>] kthread+0xd8/0xf0
[ 2817.608604] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2817.609404] [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[ 2817.610206] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
8. systemd journal and logind and irqbalanced is blocked in pagefault handler
[ 2817.716663] systemd-journal D ffff8800351c7b78 0 424 1 0x00000000
[ 2817.717463] ffff8800351c7b78 ffff880094103680 ffffffff81e11500 ffff880035458000
[ 2817.718269] ffff8800351c8000 ffff88015ec16d00 7fffffffffffffff ffffffff81821a00
[ 2817.719077] ffff8800351c7cd8 ffff8800351c7b90 ffffffff81821205 0000000000000000
[ 2817.719877] Call Trace:
[ 2817.720663] [<ffffffff81821a00>] ? bit_wait+0x60/0x60
[ 2817.721458] [<ffffffff81821205>] schedule+0x35/0x80
[ 2817.722247] [<ffffffff81824325>] schedule_timeout+0x1b5/0x270
[ 2817.723049] [<ffffffff813b5cf3>] ? __blk_run_queue+0x33/0x40
[ 2817.723852] [<ffffffff813b5fba>] ? queue_unplugged+0x2a/0xb0
[ 2817.724651] [<ffffffff81821a00>] ? bit_wait+0x60/0x60
[ 2817.725448] [<ffffffff81820754>] io_schedule_timeout+0xa4/0x110
[ 2817.726244] [<ffffffff81821a1b>] bit_wait_io+0x1b/0x70
[ 2817.727021] [<ffffffff818215ad>] __wait_on_bit+0x5d/0x90
[ 2817.727800] [<ffffffff8118f79d>] wait_on_page_bit_killable+0xcd/0xf0
[ 2817.728583] [<ffffffff810c3ab0>] ? autoremove_wake_function+0x40/0x40
[ 2817.729369] [<ffffffff8118f844>] __lock_page_or_retry+0x84/0xa0
[ 2817.730143] [<ffffffff8118fa4d>] filemap_fault+0x1ed/0x3f0
[ 2817.730920] [<ffffffff811bc1f0>] __do_fault+0x50/0xe0
[ 2817.731697] [<ffffffff811bfc2b>] handle_mm_fault+0xf8b/0x1820
[ 2817.732475] [<ffffffff8125575a>] ? ep_poll+0x21a/0x3d0
[ 2817.733252] [<ffffffff8106b537>] __do_page_fault+0x197/0x400
[ 2817.734035] [<ffffffff8106b7c2>] do_page_fault+0x22/0x30
[ 2817.734811] [<ffffffff81827478>] page_fault+0x28/0x30
[ 2817.906729] systemd-logind D ffff880157b6bb78 0 857 1 0x00000000
[ 2817.907551] ffff880157b6bb78 ffffffff8118d08e ffff880159470dc0 ffff880095402940
[ 2817.908386] ffff880157b6c000 ffff88015ec56d00 7fffffffffffffff ffffffff81821a00
[ 2817.909207] ffff880157b6bcd8 ffff880157b6bb90 ffffffff81821205 0000000000000000
[ 2817.910038] Call Trace:
[ 2817.910850] [<ffffffff8118d08e>] ? find_get_entry+0x1e/0xa0
[ 2817.911666] [<ffffffff81821a00>] ? bit_wait+0x60/0x60
[ 2817.912477] [<ffffffff81821205>] schedule+0x35/0x80
[ 2817.913289] [<ffffffff81824325>] schedule_timeout+0x1b5/0x270
[ 2817.914101] [<ffffffff811c008d>] ? handle_mm_fault+0x13ed/0x1820
[ 2817.914919] [<ffffffff81821a00>] ? bit_wait+0x60/0x60
[ 2817.915731] [<ffffffff81820754>] io_schedule_timeout+0xa4/0x110
[ 2817.916546] [<ffffffff81821a1b>] bit_wait_io+0x1b/0x70
[ 2817.917353] [<ffffffff818215ad>] __wait_on_bit+0x5d/0x90
[ 2817.918151] [<ffffffff8118f79d>] wait_on_page_bit_killable+0xcd/0xf0
[ 2817.918952] [<ffffffff810c3ab0>] ? autoremove_wake_function+0x40/0x40
[ 2817.919753] [<ffffffff8118f844>] __lock_page_or_retry+0x84/0xa0
[ 2817.920557] [<ffffffff8118fa4d>] filemap_fault+0x1ed/0x3f0
[ 2817.921349] [<ffffffff811bc1f0>] __do_fault+0x50/0xe0
[ 2817.922136] [<ffffffff811bfc2b>] handle_mm_fault+0xf8b/0x1820
[ 2817.922924] [<ffffffff8125575a>] ? ep_poll+0x21a/0x3d0
[ 2817.923714] [<ffffffff8106b537>] __do_page_fault+0x197/0x400
[ 2817.924503] [<ffffffff8106b7c2>] do_page_fault+0x22/0x30
[ 2817.925295] [<ffffffff81827478>] page_fault+0x28/0x30
[ 2818.096501] irqbalance D ffff880153413b78 0 1005 1 0x00000000
[ 2818.097266] ffff880153413b78 ffff880094347080 ffff880159472940 ffff880095406040
[ 2818.098039] ffff880153414000 ffff88015ecd6d00 7fffffffffffffff ffffffff81821a00
[ 2818.098814] ffff880153413cd8 ffff880153413b90 ffffffff81821205 0000000000000000
[ 2818.099591] Call Trace:
[ 2818.100346] [<ffffffff81821a00>] ? bit_wait+0x60/0x60
[ 2818.101114] [<ffffffff81821205>] schedule+0x35/0x80
[ 2818.101871] [<ffffffff81824325>] schedule_timeout+0x1b5/0x270
[ 2818.102626] [<ffffffff813b5cf3>] ? __blk_run_queue+0x33/0x40
[ 2818.103376] [<ffffffff813b5fba>] ? queue_unplugged+0x2a/0xb0
[ 2818.104147] [<ffffffff81821a00>] ? bit_wait+0x60/0x60
[ 2818.104894] [<ffffffff81820754>] io_schedule_timeout+0xa4/0x110
[ 2818.105638] [<ffffffff81821a1b>] bit_wait_io+0x1b/0x70
[ 2818.106392] [<ffffffff818215ad>] __wait_on_bit+0x5d/0x90
[ 2818.107142] [<ffffffff8118f79d>] wait_on_page_bit_killable+0xcd/0xf0
[ 2818.107919] [<ffffffff810c3ab0>] ? autoremove_wake_function+0x40/0x40
[ 2818.108674] [<ffffffff8118f844>] __lock_page_or_retry+0x84/0xa0
[ 2818.109435] [<ffffffff8118fa4d>] filemap_fault+0x1ed/0x3f0
[ 2818.110196] [<ffffffff811bc1f0>] __do_fault+0x50/0xe0
[ 2818.110956] [<ffffffff811bfc2b>] handle_mm_fault+0xf8b/0x1820
[ 2818.111729] [<ffffffff8106b537>] __do_page_fault+0x197/0x400
[ 2818.112489] [<ffffffff8106b7c2>] do_page_fault+0x22/0x30
[ 2818.113254] [<ffffffff81827478>] page_fault+0x28/0x30
9. avahi-daemon is blocked in pagefault handler with different
stacktrace
[ 2818.403217] avahi-daemon D ffff880090e671f8 0 3601 1 0x00000000
[ 2818.404027] ffff880090e671f8 ffffffff811eaac6 ffff8800350ce040 ffff880094910000
[ 2818.404832] ffff880090e68000 ffff880090e67230 ffff88015eccdd00 ffff88015eccdd00
[ 2818.405629] ffff880094910000 ffff880090e67210 ffffffff81821205 0000000100099ee5
[ 2818.406422] Call Trace:
[ 2818.407195] [<ffffffff811eaac6>] ? ___slab_alloc+0x2a6/0x460
[ 2818.407992] [<ffffffff81821205>] schedule+0x35/0x80
[ 2818.408775] [<ffffffff81824299>] schedule_timeout+0x129/0x270
[ 2818.409558] [<ffffffff810ec370>] ? trace_event_raw_event_tick_stop+0x120/0x120
[ 2818.410349] [<ffffffff810f574c>] ? ktime_get+0x3c/0xb0
[ 2818.411143] [<ffffffff81820754>] io_schedule_timeout+0xa4/0x110
[ 2818.411957] [<ffffffff811900c8>] mempool_alloc+0x148/0x170
[ 2818.412750] [<ffffffff810c3a70>] ? wake_atomic_t_function+0x60/0x60
[ 2818.413549] [<ffffffff813b20ad>] bio_alloc_bioset+0xbd/0x260
[ 2818.414342] [<ffffffff816a33cf>] __split_and_process_bio+0x1ef/0x3e0
[ 2818.415137] [<ffffffff816a3629>] dm_make_request+0x69/0xc0
[ 2818.415937] [<ffffffff813ba502>] generic_make_request+0xf2/0x1d0
[ 2818.416724] [<ffffffff813ba656>] submit_bio+0x76/0x170
[ 2818.417507] [<ffffffff811d2548>] __swap_writepage+0x228/0x270
[ 2818.418288] [<ffffffff811d7bcf>] ? __frontswap_store+0xdf/0x110
[ 2818.419067] [<ffffffff811d25c9>] swap_writepage+0x39/0x70
[ 2818.419857] [<ffffffff811a0b6d>] pageout.isra.44+0x16d/0x280
[ 2818.420639] [<ffffffff811a31ca>] shrink_page_list+0x3ca/0x7a0
[ 2818.421428] [<ffffffff811a3c39>] shrink_inactive_list+0x209/0x520
[ 2818.422212] [<ffffffff811a48d3>] shrink_lruvec+0x593/0x750
[ 2818.422997] [<ffffffff811a4b7f>] shrink_zone+0xef/0x2e0
[ 2818.423779] [<ffffffff811a4edc>] do_try_to_free_pages+0x16c/0x3f0
[ 2818.424558] [<ffffffff811a522e>] try_to_free_pages+0xce/0x180
[ 2818.425327] [<ffffffff81196eda>] __alloc_pages_nodemask+0x64a/0xb60
[ 2818.426100] [<ffffffff811e091c>] alloc_pages_current+0x8c/0x110
[ 2818.426869] [<ffffffff8118d77b>] __page_cache_alloc+0xab/0xc0
[ 2818.427652] [<ffffffff8119bf8b>] __do_page_cache_readahead+0xeb/0x230
[ 2818.428415] [<ffffffff8118d08e>] ? find_get_entry+0x1e/0xa0
[ 2818.429172] [<ffffffff8118fbd5>] filemap_fault+0x375/0x3f0
[ 2818.429932] [<ffffffff811bc1f0>] __do_fault+0x50/0xe0
[ 2818.430693] [<ffffffff811bfc2b>] handle_mm_fault+0xf8b/0x1820
[ 2818.431461] [<ffffffff81220f60>] ? poll_select_copy_remaining+0x140/0x140
[ 2818.432246] [<ffffffff8106b537>] __do_page_fault+0x197/0x400
[ 2818.433015] [<ffffffff8106b7c2>] do_page_fault+0x22/0x30
[ 2818.433782] [<ffffffff81827478>] page_fault+0x28/0x30
10. Memory hungry compilation job is blocked in pagefault handler with
similar trace to avahi
[ 2818.535321] cc1plus D ffff880090bcb388 0 3977 3976 0x00000000
[ 2818.536087] ffff880090bcb388 ffffffff811eaac6 ffffffff81e11500 ffff880094915280
[ 2818.536853] ffff880090bcc000 ffff880090bcb3c0 ffff88015ec0dd00 ffff88015ec0dd00
[ 2818.537618] ffff880094915280 ffff880090bcb3a0 ffffffff81821205 0000000100099ee5
[ 2818.538375] Call Trace:
[ 2818.539128] [<ffffffff811eaac6>] ? ___slab_alloc+0x2a6/0x460
[ 2818.539897] [<ffffffff81821205>] schedule+0x35/0x80
[ 2818.540665] [<ffffffff81824299>] schedule_timeout+0x129/0x270
[ 2818.541431] [<ffffffff810ec370>] ? trace_event_raw_event_tick_stop+0x120/0x120
[ 2818.542201] [<ffffffff810f574c>] ? ktime_get+0x3c/0xb0
[ 2818.542970] [<ffffffff81820754>] io_schedule_timeout+0xa4/0x110
[ 2818.543748] [<ffffffff811900c8>] mempool_alloc+0x148/0x170
[ 2818.544517] [<ffffffff810c3a70>] ? wake_atomic_t_function+0x60/0x60
[ 2818.545286] [<ffffffff813b20ad>] bio_alloc_bioset+0xbd/0x260
[ 2818.546063] [<ffffffff816a33cf>] __split_and_process_bio+0x1ef/0x3e0
[ 2818.546840] [<ffffffff816a3629>] dm_make_request+0x69/0xc0
[ 2818.547632] [<ffffffff813ba502>] generic_make_request+0xf2/0x1d0
[ 2818.548416] [<ffffffff813ba656>] submit_bio+0x76/0x170
[ 2818.549200] [<ffffffff811d2548>] __swap_writepage+0x228/0x270
[ 2818.549980] [<ffffffff811d7bcf>] ? __frontswap_store+0xdf/0x110
[ 2818.550763] [<ffffffff811d25c9>] swap_writepage+0x39/0x70
[ 2818.551546] [<ffffffff811a0b6d>] pageout.isra.44+0x16d/0x280
[ 2818.552341] [<ffffffff811a31ca>] shrink_page_list+0x3ca/0x7a0
[ 2818.553121] [<ffffffff811a3c39>] shrink_inactive_list+0x209/0x520
[ 2818.553902] [<ffffffff811a48d3>] shrink_lruvec+0x593/0x750
[ 2818.554677] [<ffffffff810986d1>] ? queue_work_on+0x31/0x40
[ 2818.555460] [<ffffffff810986d1>] ? queue_work_on+0x31/0x40
[ 2818.556230] [<ffffffff811a4b7f>] shrink_zone+0xef/0x2e0
[ 2818.557004] [<ffffffff811a4edc>] do_try_to_free_pages+0x16c/0x3f0
[ 2818.557784] [<ffffffff811a522e>] try_to_free_pages+0xce/0x180
[ 2818.558561] [<ffffffff81196eda>] __alloc_pages_nodemask+0x64a/0xb60
[ 2818.559337] [<ffffffff811e216e>] alloc_pages_vma+0xbe/0x240
[ 2818.560123] [<ffffffff811c012e>] handle_mm_fault+0x148e/0x1820
[ 2818.560884] [<ffffffff8106b537>] __do_page_fault+0x197/0x400
[ 2818.561643] [<ffffffff8106b7c2>] do_page_fault+0x22/0x30
[ 2818.562397] [<ffffffff81827478>] page_fault+0x28/0x30
11. disk events work is blocked in lru shrink
[ 2818.563150] kworker/u16:5 D ffff8800958135c8 0 3978 2 0x00000000
[ 2818.563922] Workqueue: events_freezable_power_ disk_events_workfn
[ 2818.564681] ffff8800958135c8 ffffffff811cc3a6 ffffffff81e11500 ffff8800350ce040
[ 2818.565456] ffff880095814000 ffff880095813600 ffff88015ec0dd00 ffff88015ec0dd00
[ 2818.566234] ffffffffffffffe0 ffff8800958135e0 ffffffff81821205 0000000100099b6b
[ 2818.567022] Call Trace:
[ 2818.567803] [<ffffffff811cc3a6>] ? page_referenced+0xb6/0x140
[ 2818.568593] [<ffffffff81821205>] schedule+0x35/0x80
[ 2818.569372] [<ffffffff81824299>] schedule_timeout+0x129/0x270
[ 2818.570152] [<ffffffff810ec370>] ? trace_event_raw_event_tick_stop+0x120/0x120
[ 2818.570936] [<ffffffff8182443e>] schedule_timeout_uninterruptible+0x1e/0x20
[ 2818.571720] [<ffffffff811aea13>] wait_iff_congested+0xf3/0x180
[ 2818.572505] [<ffffffff810c3a70>] ? wake_atomic_t_function+0x60/0x60
[ 2818.573283] [<ffffffff811a3f2f>] shrink_inactive_list+0x4ff/0x520
[ 2818.574056] [<ffffffff811a48d3>] shrink_lruvec+0x593/0x750
[ 2818.574829] [<ffffffff8120ebc0>] ? super_cache_count+0x70/0xe0
[ 2818.575610] [<ffffffff811a4b7f>] shrink_zone+0xef/0x2e0
[ 2818.576381] [<ffffffff811a4edc>] do_try_to_free_pages+0x16c/0x3f0
[ 2818.577158] [<ffffffff811a522e>] try_to_free_pages+0xce/0x180
[ 2818.577941] [<ffffffff81196eda>] __alloc_pages_nodemask+0x64a/0xb60
[ 2818.578717] [<ffffffff813b60b9>] ? alloc_request_struct+0x19/0x20
[ 2818.579488] [<ffffffff811e091c>] alloc_pages_current+0x8c/0x110
[ 2818.580262] [<ffffffff813b33e7>] bio_copy_kern+0xd7/0x1e0
[ 2818.581030] [<ffffffff813c0088>] blk_rq_map_kern+0xa8/0x140
[ 2818.581797] [<ffffffff813b9e5b>] ? blk_get_request+0x8b/0xf0
[ 2818.582558] [<ffffffff815adc68>] scsi_execute+0x78/0x1d0
[ 2818.583317] [<ffffffff815af96e>] scsi_execute_req_flags+0x8e/0xf0
[ 2818.584087] [<ffffffff815c0e27>] sr_check_events+0xb7/0x2d0
[ 2818.584845] [<ffffffff815fe0d8>] cdrom_check_events+0x18/0x30
[ 2818.585610] [<ffffffff815c125a>] sr_block_check_events+0x2a/0x30
[ 2818.586367] [<ffffffff813cb5f0>] disk_check_events+0x60/0x150
[ 2818.587119] [<ffffffff813cb6f6>] disk_events_workfn+0x16/0x20
[ 2818.587875] [<ffffffff8109a052>] process_one_work+0x162/0x480
[ 2818.588624] [<ffffffff8109a3bb>] worker_thread+0x4b/0x4c0
[ 2818.589353] [<ffffffff8109a370>] ? process_one_work+0x480/0x480
[ 2818.590090] [<ffffffff8109a370>] ? process_one_work+0x480/0x480
[ 2818.590817] [<ffffffff810a0588>] kthread+0xd8/0xf0
[ 2818.591535] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2818.592274] [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[ 2818.592993] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
12. kernel workqueues have a lot of stuf waiting execution
[ 2819.710760]
[ 2819.711509] Showing busy workqueues and worker pools:
[ 2819.712278] workqueue events_freezable_power_: flags=0x86
[ 2819.713053] pwq 16: cpus=0-7 flags=0x4 nice=0 active=1/256
[ 2819.713843] in-flight: 3978:disk_events_workfn
[ 2819.714673] workqueue kcryptd: flags=0x2a
[ 2819.715464] pwq 16: cpus=0-7 flags=0x4 nice=0 active=4/4
[ 2819.716267] in-flight: 3966:kcryptd_crypt [dm_crypt], 160:kcryptd_crypt [dm_crypt], 3388:kcryptd_crypt [dm_crypt], 3339:kcryptd_crypt [dm_crypt]
[ 2819.717131] delayed: kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], <repeats enought times to make line 17983 characters wide>
[ 2822.339640] pool 16: cpus=0-7 flags=0x4 nice=0 workers=7 idle: 3979 3139
13. Swap usage stats from kernel:
[ 2960.187650] Swap cache stats: add 16096, delete 3484, find 193/223
[ 2960.187711] Free swap = 4054404kB
[ 2960.187748] Total swap = 4112380kB
Recovery from hang state:
Sometimes hang state can be recovred if I use sysrq-k and sysrq-e to kill all userspace programs. Systemd will then restart userspace process if hanged state was recovered successfully. Recovering doesn't work
Workaround:
Disabling swap with swapoff -a.
Running same compile job results to OOM killer cleanly killing the memory hungry process.
ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-4.4.0-22-generic 4.4.0-22.40
ProcVersionSignature: Ubuntu 4.4.0-22.40-generic 4.4.8
Uname: Linux 4.4.0-22-generic x86_64
NonfreeKernelModules: nvidia
ApportVersion: 2.20.1-0ubuntu2.1
Architecture: amd64
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path', '/dev/snd/hwC1D0', '/dev/snd/pcmC1D0c', '/dev/snd/pcmC1D0p', '/dev/snd/controlC1', '/dev/snd/hwC0D0', '/dev/snd/pcmC0D8p', '/dev/snd/pcmC0D7p', '/dev/snd/pcmC0D3p', '/dev/snd/controlC0', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CurrentDesktop: GNOME
Date: Mon Jun 6 17:55:26 2016
HibernationDevice: RESUME=UUID=04553608-429c-40dc-b53c-a200f2a70588
InstallationDate: Installed on 2037-12-25 (-7871 days ago)
InstallationMedia: Lubuntu 16.04 LTS "Xenial Xerus" - Release amd64 (20160420.1)
MachineType: Acer Aspire E5-571G
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-22-generic root=/dev/mapper/lubuntu--vg-root ro quiet splash
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
linux-restricted-modules-4.4.0-22-generic N/A
linux-backports-modules-4.4.0-22-generic N/A
linux-firmware 1.157
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 09/15/2015
dmi.bios.vendor: Insyde Corp.
dmi.bios.version: V1.32
dmi.board.name: EA50_HB
dmi.board.vendor: Acer
dmi.board.version: V1.32
dmi.chassis.type: 10
dmi.chassis.vendor: Acer
dmi.chassis.version: V1.32
dmi.modalias: dmi:bvnInsydeCorp.:bvrV1.32:bd09/15/2015:svnAcer:pnAspireE5-571G:pvrV1.32:rvnAcer:rnEA50_HB:rvrV1.32:cvnAcer:ct10:cvrV1.32:
dmi.product.name: Aspire E5-571G
dmi.product.version: V1.32
dmi.sys.vendor: Acer
** Affects: linux (Ubuntu)
Importance: Undecided
Status: Confirmed
** Tags: amd64 apport-bug xenial
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1589563
Title:
ext4 journaling and swapping to same encrypted SSHD hangs system
Status in linux package in Ubuntu:
Confirmed
Bug description:
Short description:
The system has single SSHD disk with luks/lvm2 encrypted root file system and swap partition. If system starts to swap ext4 journal commit and swap requests hang waiting disk io.
Relevant bug reports in other places:
So far I have managed to find only one relevant looking report about very similar problem
https://bbs.archlinux.org/viewtopic.php?id=193034
Hardware:
Acer Aspire E15 E5-571G59EQ
Intel Broadwell i5-5200U
4GB ram
nvidia GeForce 840M (binary driver installed but disabled)
Seagate ST500LM000-1EJ16 500GB HDD + 8GB SSHD
Known ways to reproduce:
Compiling templated c++ code using g++ 5 that results to g++ optimization taking huge amount of memory.
I haven't yet tested if there is any other
Reproduction details:
$ cat main.cpp
#include <iostream>
template<unsigned idx>
unsigned loop()
{
if (idx == 1)
return 6;
unsigned count = 0;
for (unsigned i = 1; i < 7; i++) {
count += loop< (idx > 1 ? idx - 1 : 1) >();
}
return count;
}
int main(void)
{
unsigned count, dice[5];
std::cout << "6^5: " << 6*6*6*6*6 << "\n";
std::cout << "loop all: " << loop<5>() << "\n";
std::cout << "loop all two players: " << loop<10>() << "\n";
return 0;
}
$ gcc --version
gcc (Ubuntu 5.3.1-14ubuntu2.1) 5.3.1 20160413
$ g++ -O2 -g -c -o main.o main.cpp
Relevant backtraces from sysrq-t (recorded using netconsole):
1. Watchdog catches jdb2 hang
[ 2784.570413] INFO: task jbd2/dm-1-8:372 blocked for more than 10 seconds.
[ 2784.571209] Tainted: P OE 4.4.0-22-generic #40-Ubuntu
[ 2784.571948] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2784.572606] jbd2/dm-1-8 D ffff880035577ad8 0 372 2 0x00000000
[ 2784.573294] ffff880035577ad8 ffff880035577b00 ffff880159472940 ffff8801561e0dc0
[ 2784.573997] ffff880035578000 ffff88015ecd6d00 7fffffffffffffff ffffffff81821a00
[ 2784.574724] ffff880035577c30 ffff880035577af0 ffffffff81821205 0000000000000000
[ 2784.575445] Call Trace:
[ 2784.576124] [<ffffffff81821a00>] ? bit_wait+0x60/0x60
[ 2784.576835] [<ffffffff81821205>] schedule+0x35/0x80
[ 2784.577546] [<ffffffff81824325>] schedule_timeout+0x1b5/0x270
[ 2784.578255] [<ffffffff813b5cf3>] ? __blk_run_queue+0x33/0x40
[ 2784.578959] [<ffffffff813bc3d0>] ? blk_queue_bio+0x3d0/0x3e0
[ 2784.579635] [<ffffffff810f574c>] ? ktime_get+0x3c/0xb0
[ 2784.580328] [<ffffffff81821a00>] ? bit_wait+0x60/0x60
[ 2784.581047] [<ffffffff81820754>] io_schedule_timeout+0xa4/0x110
[ 2784.581767] [<ffffffff81821a1b>] bit_wait_io+0x1b/0x70
[ 2784.582509] [<ffffffff818215ad>] __wait_on_bit+0x5d/0x90
[ 2784.583215] [<ffffffff81821a00>] ? bit_wait+0x60/0x60
[ 2784.583951] [<ffffffff81821662>] out_of_line_wait_on_bit+0x82/0xb0
[ 2784.584687] [<ffffffff810c3ab0>] ? autoremove_wake_function+0x40/0x40
[ 2784.585434] [<ffffffff81243c32>] __wait_on_buffer+0x32/0x40
[ 2784.586174] [<ffffffff812eaf6f>] jbd2_journal_commit_transaction+0x10cf/0x1870
[ 2784.586906] [<ffffffff810ec5fe>] ? try_to_del_timer_sync+0x5e/0x90
[ 2784.587622] [<ffffffff812ef32a>] kjournald2+0xca/0x250
[ 2784.588339] [<ffffffff810c3a70>] ? wake_atomic_t_function+0x60/0x60
[ 2784.589058] [<ffffffff812ef260>] ? commit_timeout+0x10/0x10
[ 2784.589769] [<ffffffff810a0588>] kthread+0xd8/0xf0
[ 2784.590503] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2784.591215] [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[ 2784.591930] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
2. kswaped looks like hanged too
[ 2817.029410] kswapd0 D ffff8800934ab608 0 43 2 0x00000000
[ 2817.030212] ffff8800934ab608 ffffffff811eaac6 ffff880093b6b700 ffff8800934a0000
[ 2817.031032] ffff8800934ac000 ffff8800934ab640 ffff88015eccdd00 ffff88015eccdd00
[ 2817.031860] ffff8800934a0000 ffff8800934ab620 ffffffff81821205 0000000100099a03
[ 2817.032670] Call Trace:
[ 2817.033473] [<ffffffff811eaac6>] ? ___slab_alloc+0x2a6/0x460
[ 2817.034282] [<ffffffff81821205>] schedule+0x35/0x80
[ 2817.035082] [<ffffffff81824299>] schedule_timeout+0x129/0x270
[ 2817.035898] [<ffffffff810ec370>] ? trace_event_raw_event_tick_stop+0x120/0x120
[ 2817.036709] [<ffffffff810f574c>] ? ktime_get+0x3c/0xb0
[ 2817.037515] [<ffffffff81820754>] io_schedule_timeout+0xa4/0x110
[ 2817.038327] [<ffffffff811900c8>] mempool_alloc+0x148/0x170
[ 2817.039137] [<ffffffff810c3a70>] ? wake_atomic_t_function+0x60/0x60
[ 2817.039964] [<ffffffff813b20ad>] bio_alloc_bioset+0xbd/0x260
[ 2817.040774] [<ffffffff816a33cf>] __split_and_process_bio+0x1ef/0x3e0
[ 2817.041576] [<ffffffff816a3629>] dm_make_request+0x69/0xc0
[ 2817.042377] [<ffffffff813ba502>] generic_make_request+0xf2/0x1d0
[ 2817.043180] [<ffffffff813ba656>] submit_bio+0x76/0x170
[ 2817.043999] [<ffffffff811d2548>] __swap_writepage+0x228/0x270
[ 2817.044799] [<ffffffff811d7bcf>] ? __frontswap_store+0xdf/0x110
[ 2817.045589] [<ffffffff811d25c9>] swap_writepage+0x39/0x70
[ 2817.046386] [<ffffffff811a0b6d>] pageout.isra.44+0x16d/0x280
[ 2817.047179] [<ffffffff811a31ca>] shrink_page_list+0x3ca/0x7a0
[ 2817.047990] [<ffffffff811a3c39>] shrink_inactive_list+0x209/0x520
[ 2817.048785] [<ffffffff811a48d3>] shrink_lruvec+0x593/0x750
[ 2817.049582] [<ffffffff81098239>] ? __queue_work+0x139/0x3b0
[ 2817.050373] [<ffffffff811a4b7f>] shrink_zone+0xef/0x2e0
[ 2817.051159] [<ffffffff811a5dfe>] kswapd+0x51e/0x990
[ 2817.051952] [<ffffffff811a58e0>] ? mem_cgroup_shrink_node_zone+0x1c0/0x1c0
[ 2817.052731] [<ffffffff810a0588>] kthread+0xd8/0xf0
[ 2817.053512] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2817.054299] [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[ 2817.055083] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
3. encyptfs-thread and bioset looks like sleeping without work
[ 2817.076543] ecryptfs-kthrea S ffff880093553e40 0 46 2 0x00000000
[ 2817.077347] ffff880093553e40 00000000934a29a0 ffff880159472940 ffff8800934a2940
[ 2817.078160] ffff880093554000 ffff880093553e80 ffff8800934a2940 0000000000000000
[ 2817.078980] 0000000000000000 ffff880093553e58 ffffffff81821205 0000000000000000
[ 2817.079795] Call Trace:
[ 2817.080596] [<ffffffff81821205>] schedule+0x35/0x80
[ 2817.081407] [<ffffffff8130b02a>] ecryptfs_threadfn+0x17a/0x1c0
[ 2817.082212] [<ffffffff810c3a70>] ? wake_atomic_t_function+0x60/0x60
[ 2817.083017] [<ffffffff8130aeb0>] ? ecryptfs_add_global_auth_tok+0xa0/0xa0
[ 2817.083829] [<ffffffff810a0588>] kthread+0xd8/0xf0
[ 2817.084628] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2817.085432] [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[ 2817.086227] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
4. scsi threads seem to be sleeping (for all cores same)
[ 2817.390470] scsi_eh_0 S ffff8800357c7e10 0 152 2 0x00000000
[ 2817.391287] ffff8800357c7e10 ffff8800357c7e08 ffff880159472940 ffff8800937a5280
[ 2817.392107] ffff8800357c8000 ffff8800937a5858 ffff8800937a5858 ffff8800937a5280
[ 2817.392917] 0000000000000000 ffff8800357c7e28 ffffffff81821205 ffff8800356c9800
[ 2817.393730] Call Trace:
[ 2817.394537] [<ffffffff81821205>] schedule+0x35/0x80
[ 2817.395339] [<ffffffff815ad2f7>] scsi_error_handler+0x97/0x8a0
[ 2817.396142] [<ffffffff81820b46>] ? __schedule+0x386/0xa10
[ 2817.396946] [<ffffffff815ad260>] ? scsi_eh_get_sense+0x250/0x250
[ 2817.397748] [<ffffffff810a0588>] kthread+0xd8/0xf0
[ 2817.398553] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2817.399363] [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[ 2817.400170] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2817.400977] scsi_tmf_0 S ffff8800357cbe38 0 153 2 0x00000000
[ 2817.401800] ffff8800357cbe38 ffff8800357cbe40 ffff880159470dc0 ffff8800937a44c0
[ 2817.402624] ffff8800357cc000 ffff8800937a44c0 ffffffff8109a830 ffff8800937e9a80
[ 2817.403439] ffff8800356e0000 ffff8800357cbe50 ffffffff81821205 ffff8800356e0030
[ 2817.404257] Call Trace:
[ 2817.405064] [<ffffffff8109a830>] ? worker_thread+0x4c0/0x4c0
[ 2817.405870] [<ffffffff81821205>] schedule+0x35/0x80
[ 2817.406673] [<ffffffff8109ab75>] rescuer_thread+0x345/0x3d0
[ 2817.407475] [<ffffffff8109a830>] ? worker_thread+0x4c0/0x4c0
[ 2817.408279] [<ffffffff810a0588>] kthread+0xd8/0xf0
[ 2817.409083] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2817.409889] [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[ 2817.410699] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
5. kworker running dm_crypt work seem to be waiting for something in
lru shrink
[ 2817.474917] kworker/u16:3 R running task 0 160 2 0x00000000
[ 2817.475748] Workqueue: kcryptd kcryptd_crypt [dm_crypt]
[ 2817.476566] ffff880035007538 ffffffff811cc3a6 ffff88015723b700 ffff8800354044c0
[ 2817.477396] ffff880035008000 ffff880035007570 ffff88015ec0dd00 ffff88015ec0dd00
[ 2817.478218] ffffffffffffffe0 ffff880035007550 ffffffff81821205 0000000100099990
[ 2817.479052] Call Trace:
[ 2817.479866] [<ffffffff811cc3a6>] ? page_referenced+0xb6/0x140
[ 2817.480694] [<ffffffff81821205>] schedule+0x35/0x80
[ 2817.481520] [<ffffffff81824299>] schedule_timeout+0x129/0x270
[ 2817.482337] [<ffffffff810ec370>] ? trace_event_raw_event_tick_stop+0x120/0x120
[ 2817.483173] [<ffffffff8182443e>] schedule_timeout_uninterruptible+0x1e/0x20
[ 2817.484009] [<ffffffff811aea13>] wait_iff_congested+0xf3/0x180
[ 2817.484847] [<ffffffff810c3a70>] ? wake_atomic_t_function+0x60/0x60
[ 2817.485682] [<ffffffff811a3f2f>] shrink_inactive_list+0x4ff/0x520
[ 2817.486513] [<ffffffff811a48d3>] shrink_lruvec+0x593/0x750
[ 2817.487332] [<ffffffff811a4b7f>] shrink_zone+0xef/0x2e0
[ 2817.488151] [<ffffffff811a4edc>] do_try_to_free_pages+0x16c/0x3f0
[ 2817.488978] [<ffffffff811a522e>] try_to_free_pages+0xce/0x180
[ 2817.489795] [<ffffffff81196eda>] __alloc_pages_nodemask+0x64a/0xb60
[ 2817.490608] [<ffffffff811ade4c>] ? zone_statistics+0x7c/0xa0
[ 2817.491423] [<ffffffff811e091c>] alloc_pages_current+0x8c/0x110
[ 2817.492237] [<ffffffff811e99ff>] new_slab+0x28f/0x490
[ 2817.493050] [<ffffffff811eaa4b>] ___slab_alloc+0x22b/0x460
[ 2817.493864] [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
[ 2817.494663] [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
[ 2817.495466] [<ffffffff811eaca0>] __slab_alloc+0x20/0x40
[ 2817.496264] [<ffffffff811eb21f>] kmem_cache_alloc+0x19f/0x1f0
[ 2817.497065] [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
[ 2817.497858] [<ffffffff8118ff05>] mempool_alloc_slab+0x15/0x20
[ 2817.498644] [<ffffffff8118ffee>] mempool_alloc+0x6e/0x170
[ 2817.499429] [<ffffffffc0162092>] ? __ablk_encrypt+0x52/0x70 [ablk_helper]
[ 2817.500218] [<ffffffff813b20ad>] bio_alloc_bioset+0xbd/0x260
[ 2817.501003] [<ffffffffc0112af4>] kcryptd_crypt+0x104/0x380 [dm_crypt]
[ 2817.501787] [<ffffffff8109a052>] process_one_work+0x162/0x480
[ 2817.502572] [<ffffffff8109a3bb>] worker_thread+0x4b/0x4c0
[ 2817.503356] [<ffffffff8109a370>] ? process_one_work+0x480/0x480
[ 2817.504135] [<ffffffff810a0588>] kthread+0xd8/0xf0
[ 2817.504919] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2817.505704] [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[ 2817.506490] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2818.290300] kworker/u16:1 D ffff8801561fb538 0 3339 2 0x00000000
[ 2818.291008] Workqueue: kcryptd kcryptd_crypt [dm_crypt]
[ 2818.291734] ffff8801561fb538 ffffffff811cc3a6 ffff880159472940 ffff880093b82940
[ 2818.292466] ffff8801561fc000 ffff8801561fb570 ffff88015eccdd00 ffff88015eccdd00
[ 2818.293201] ffffffffffffffe0 ffff8801561fb550 ffffffff81821205 0000000100099b27
[ 2818.293938] Call Trace:
[ 2818.294658] [<ffffffff811cc3a6>] ? page_referenced+0xb6/0x140
[ 2818.295400] [<ffffffff81821205>] schedule+0x35/0x80
[ 2818.296162] [<ffffffff81824299>] schedule_timeout+0x129/0x270
[ 2818.296924] [<ffffffff810ec370>] ? trace_event_raw_event_tick_stop+0x120/0x120
[ 2818.297700] [<ffffffff8182443e>] schedule_timeout_uninterruptible+0x1e/0x20
[ 2818.298479] [<ffffffff811aea13>] wait_iff_congested+0xf3/0x180
[ 2818.299264] [<ffffffff810c3a70>] ? wake_atomic_t_function+0x60/0x60
[ 2818.300063] [<ffffffff811a3f2f>] shrink_inactive_list+0x4ff/0x520
[ 2818.300853] [<ffffffff811a48d3>] shrink_lruvec+0x593/0x750
[ 2818.301645] [<ffffffff811a4b7f>] shrink_zone+0xef/0x2e0
[ 2818.302437] [<ffffffff811a4edc>] do_try_to_free_pages+0x16c/0x3f0
[ 2818.303232] [<ffffffff811a522e>] try_to_free_pages+0xce/0x180
[ 2818.304038] [<ffffffff81196eda>] __alloc_pages_nodemask+0x64a/0xb60
[ 2818.304831] [<ffffffff811ade4c>] ? zone_statistics+0x7c/0xa0
[ 2818.305619] [<ffffffff811e091c>] alloc_pages_current+0x8c/0x110
[ 2818.306399] [<ffffffff811e99ff>] new_slab+0x28f/0x490
[ 2818.307183] [<ffffffff811eaa4b>] ___slab_alloc+0x22b/0x460
[ 2818.307975] [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
[ 2818.308751] [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
[ 2818.309513] [<ffffffff811eaca0>] __slab_alloc+0x20/0x40
[ 2818.310279] [<ffffffff811eb21f>] kmem_cache_alloc+0x19f/0x1f0
[ 2818.311049] [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
[ 2818.311834] [<ffffffff8118ff05>] mempool_alloc_slab+0x15/0x20
[ 2818.312603] [<ffffffff8118ffee>] mempool_alloc+0x6e/0x170
[ 2818.313377] [<ffffffffc0162092>] ? __ablk_encrypt+0x52/0x70 [ablk_helper]
[ 2818.314150] [<ffffffff813b20ad>] bio_alloc_bioset+0xbd/0x260
[ 2818.314925] [<ffffffffc0112af4>] kcryptd_crypt+0x104/0x380 [dm_crypt]
[ 2818.315712] [<ffffffff8109a052>] process_one_work+0x162/0x480
[ 2818.316484] [<ffffffff8109a3bb>] worker_thread+0x4b/0x4c0
[ 2818.317250] [<ffffffff8109a370>] ? process_one_work+0x480/0x480
[ 2818.318018] [<ffffffff810a0588>] kthread+0xd8/0xf0
[ 2818.318782] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2818.319561] [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[ 2818.320333] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2818.321113] kworker/u16:2 D ffff880098653538 0 3388 2 0x00000000
[ 2818.321909] Workqueue: kcryptd kcryptd_crypt [dm_crypt]
[ 2818.322700] ffff880098653538 ffffffff811cc3a6 ffff880159471b80 ffff880093b6b700
[ 2818.323508] ffff880098654000 ffff880098653570 ffff88015ec8dd00 ffff88015ec8dd00
[ 2818.324313] ffffffffffffffe0 ffff880098653550 ffffffff81821205 0000000100099990
[ 2818.325116] Call Trace:
[ 2818.325905] [<ffffffff811cc3a6>] ? page_referenced+0xb6/0x140
[ 2818.326705] [<ffffffff81821205>] schedule+0x35/0x80
[ 2818.327498] [<ffffffff81824299>] schedule_timeout+0x129/0x270
[ 2818.328301] [<ffffffff810ec370>] ? trace_event_raw_event_tick_stop+0x120/0x120
[ 2818.329103] [<ffffffff8182443e>] schedule_timeout_uninterruptible+0x1e/0x20
[ 2818.329908] [<ffffffff811aea13>] wait_iff_congested+0xf3/0x180
[ 2818.330711] [<ffffffff810c3a70>] ? wake_atomic_t_function+0x60/0x60
[ 2818.331516] [<ffffffff811a3f2f>] shrink_inactive_list+0x4ff/0x520
[ 2818.332332] [<ffffffff811a48d3>] shrink_lruvec+0x593/0x750
[ 2818.333128] [<ffffffff811a4b7f>] shrink_zone+0xef/0x2e0
[ 2818.333912] [<ffffffff811a4edc>] do_try_to_free_pages+0x16c/0x3f0
[ 2818.334698] [<ffffffff811a522e>] try_to_free_pages+0xce/0x180
[ 2818.335483] [<ffffffff81194d44>] ? drain_local_pages+0x24/0x30
[ 2818.336278] [<ffffffff81196eda>] __alloc_pages_nodemask+0x64a/0xb60
[ 2818.337060] [<ffffffff811ade4c>] ? zone_statistics+0x7c/0xa0
[ 2818.337844] [<ffffffff811e091c>] alloc_pages_current+0x8c/0x110
[ 2818.338628] [<ffffffff811e99ff>] new_slab+0x28f/0x490
[ 2818.339407] [<ffffffff811eaa4b>] ___slab_alloc+0x22b/0x460
[ 2818.340204] [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
[ 2818.340982] [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
[ 2818.341756] [<ffffffff811eaca0>] __slab_alloc+0x20/0x40
[ 2818.342517] [<ffffffff811eb21f>] kmem_cache_alloc+0x19f/0x1f0
[ 2818.343276] [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
[ 2818.344044] [<ffffffff8118ff05>] mempool_alloc_slab+0x15/0x20
[ 2818.344793] [<ffffffff8118ffee>] mempool_alloc+0x6e/0x170
[ 2818.345540] [<ffffffffc0162092>] ? __ablk_encrypt+0x52/0x70 [ablk_helper]
[ 2818.346287] [<ffffffff813b20ad>] bio_alloc_bioset+0xbd/0x260
[ 2818.347022] [<ffffffffc0112af4>] kcryptd_crypt+0x104/0x380 [dm_crypt]
[ 2818.347775] [<ffffffff8109a052>] process_one_work+0x162/0x480
[ 2818.348518] [<ffffffff8109a3bb>] worker_thread+0x4b/0x4c0
[ 2818.349260] [<ffffffff8109a370>] ? process_one_work+0x480/0x480
[ 2818.349990] [<ffffffff810a0588>] kthread+0xd8/0xf0
[ 2818.350722] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2818.351456] [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[ 2818.352199] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
2818.478112] kworker/u16:4 D ffff880094c5b538 0 3966 2 0x00000000
[ 2818.478856] Workqueue: kcryptd kcryptd_crypt [dm_crypt]
[ 2818.479610] ffff880094c5b538 ffffffff811cc3a6 ffff880159470dc0 ffff88015723b700
[ 2818.480374] ffff880094c5c000 ffff880094c5b570 ffff88015ec4dd00 ffff88015ec4dd00
[ 2818.481137] ffffffffffffffe0 ffff880094c5b550 ffffffff81821205 0000000100099b56
[ 2818.481901] Call Trace:
[ 2818.482651] [<ffffffff811cc3a6>] ? page_referenced+0xb6/0x140
[ 2818.483413] [<ffffffff81821205>] schedule+0x35/0x80
[ 2818.484186] [<ffffffff81824299>] schedule_timeout+0x129/0x270
[ 2818.484940] [<ffffffff810ec370>] ? trace_event_raw_event_tick_stop+0x120/0x120
[ 2818.485691] [<ffffffff8182443e>] schedule_timeout_uninterruptible+0x1e/0x20
[ 2818.486447] [<ffffffff811aea13>] wait_iff_congested+0xf3/0x180
[ 2818.487201] [<ffffffff810c3a70>] ? wake_atomic_t_function+0x60/0x60
[ 2818.487964] [<ffffffff811a3f2f>] shrink_inactive_list+0x4ff/0x520
[ 2818.488712] [<ffffffff811a48d3>] shrink_lruvec+0x593/0x750
[ 2818.489468] [<ffffffff811a4b7f>] shrink_zone+0xef/0x2e0
[ 2818.490209] [<ffffffff811a4edc>] do_try_to_free_pages+0x16c/0x3f0
[ 2818.490951] [<ffffffff811a522e>] try_to_free_pages+0xce/0x180
[ 2818.491711] [<ffffffff81194d44>] ? drain_local_pages+0x24/0x30
[ 2818.492459] [<ffffffff81196eda>] __alloc_pages_nodemask+0x64a/0xb60
[ 2818.493202] [<ffffffff811ade4c>] ? zone_statistics+0x7c/0xa0
[ 2818.493944] [<ffffffff811e091c>] alloc_pages_current+0x8c/0x110
[ 2818.494696] [<ffffffff811e99ff>] new_slab+0x28f/0x490
[ 2818.495448] [<ffffffff811eaa4b>] ___slab_alloc+0x22b/0x460
[ 2818.496210] [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
[ 2818.496964] [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
[ 2818.497715] [<ffffffff811eaca0>] __slab_alloc+0x20/0x40
[ 2818.498469] [<ffffffff811eb21f>] kmem_cache_alloc+0x19f/0x1f0
[ 2818.499220] [<ffffffff8118ff05>] ? mempool_alloc_slab+0x15/0x20
[ 2818.499973] [<ffffffff8118ff05>] mempool_alloc_slab+0x15/0x20
[ 2818.500708] [<ffffffff8118ffee>] mempool_alloc+0x6e/0x170
[ 2818.501437] [<ffffffffc0162092>] ? __ablk_encrypt+0x52/0x70 [ablk_helper]
[ 2818.502169] [<ffffffff813b20ad>] bio_alloc_bioset+0xbd/0x260
[ 2818.502888] [<ffffffffc0112af4>] kcryptd_crypt+0x104/0x380 [dm_crypt]
[ 2818.503624] [<ffffffff8109a052>] process_one_work+0x162/0x480
[ 2818.504346] [<ffffffff8109a3bb>] worker_thread+0x4b/0x4c0
[ 2818.505069] [<ffffffff8109a370>] ? process_one_work+0x480/0x480
[ 2818.505794] [<ffffffff8109a370>] ? process_one_work+0x480/0x480
[ 2818.506496] [<ffffffff810a0588>] kthread+0xd8/0xf0
[ 2818.507205] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2818.507930] [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[ 2818.508642] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
6. kcryptd seems to be sleeping without work
[ 2817.579092] kcryptd_io S ffff8800352ebe38 0 318 2 0x00000000
[ 2817.579901] ffff8800352ebe38 ffff8800352ebe40 ffffffff81e11500 ffff880035068dc0
[ 2817.580712] ffff8800352ec000 ffff880035068dc0 ffffffff8109a830 ffff880035686a80
[ 2817.581516] ffff8800937d9180 ffff8800352ebe50 ffffffff81821205 ffff8800937d91b0
[ 2817.582314] Call Trace:
[ 2817.583104] [<ffffffff8109a830>] ? worker_thread+0x4c0/0x4c0
[ 2817.583904] [<ffffffff81821205>] schedule+0x35/0x80
[ 2817.584699] [<ffffffff8109ab75>] rescuer_thread+0x345/0x3d0
[ 2817.585494] [<ffffffff8109a830>] ? worker_thread+0x4c0/0x4c0
[ 2817.586284] [<ffffffff810a0588>] kthread+0xd8/0xf0
[ 2817.587076] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2817.587870] [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[ 2817.588663] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2817.589463] kcryptd S ffff8800352efe38 0 319 2 0x00000000
[ 2817.590270] ffff8800352efe38 ffff8800352efe40 ffffffff81e11500 ffff880035069b80
[ 2817.591078] ffff8800352f0000 ffff880035069b80 ffffffff8109a830 ffff8800355b9480
[ 2817.591881] ffff880156062c00 ffff8800352efe50 ffffffff81821205 ffff880156062c30
[ 2817.592681] Call Trace:
[ 2817.593468] [<ffffffff8109a830>] ? worker_thread+0x4c0/0x4c0
[ 2817.594250] [<ffffffff81821205>] schedule+0x35/0x80
[ 2817.595034] [<ffffffff8109ab75>] rescuer_thread+0x345/0x3d0
[ 2817.595828] [<ffffffff8109a830>] ? worker_thread+0x4c0/0x4c0
[ 2817.596618] [<ffffffff810a0588>] kthread+0xd8/0xf0
[ 2817.597413] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2817.598214] [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[ 2817.599014] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
7. dmcrypt_write seems like sleeping without work
[ 2817.599818] dmcrypt_write S ffff8800350f3de8 0 320 2 0x00000000
[ 2817.600626] ffff8800350f3de8 ffff8800350f3da8 ffff880159470dc0 ffff88003506a940
[ 2817.601435] ffff8800350f4000 ffff8800350f3e40 ffff8800355b9670 ffff88003506a940
[ 2817.602237] ffff8800355b9600 ffff8800350f3e00 ffffffff81821205 ffff8800355b9668
[ 2817.603038] Call Trace:
[ 2817.603827] [<ffffffff81821205>] schedule+0x35/0x80
[ 2817.604618] [<ffffffffc0113081>] dmcrypt_write+0xd1/0x1e0 [dm_crypt]
[ 2817.605412] [<ffffffff810abe40>] ? wake_up_q+0x70/0x70
[ 2817.606204] [<ffffffffc0112fb0>] ? crypt_iv_lmk_gen+0xc0/0xc0 [dm_crypt]
[ 2817.607008] [<ffffffffc0112fb0>] ? crypt_iv_lmk_gen+0xc0/0xc0 [dm_crypt]
[ 2817.607807] [<ffffffff810a0588>] kthread+0xd8/0xf0
[ 2817.608604] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2817.609404] [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[ 2817.610206] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
8. systemd journal and logind and irqbalanced is blocked in pagefault handler
[ 2817.716663] systemd-journal D ffff8800351c7b78 0 424 1 0x00000000
[ 2817.717463] ffff8800351c7b78 ffff880094103680 ffffffff81e11500 ffff880035458000
[ 2817.718269] ffff8800351c8000 ffff88015ec16d00 7fffffffffffffff ffffffff81821a00
[ 2817.719077] ffff8800351c7cd8 ffff8800351c7b90 ffffffff81821205 0000000000000000
[ 2817.719877] Call Trace:
[ 2817.720663] [<ffffffff81821a00>] ? bit_wait+0x60/0x60
[ 2817.721458] [<ffffffff81821205>] schedule+0x35/0x80
[ 2817.722247] [<ffffffff81824325>] schedule_timeout+0x1b5/0x270
[ 2817.723049] [<ffffffff813b5cf3>] ? __blk_run_queue+0x33/0x40
[ 2817.723852] [<ffffffff813b5fba>] ? queue_unplugged+0x2a/0xb0
[ 2817.724651] [<ffffffff81821a00>] ? bit_wait+0x60/0x60
[ 2817.725448] [<ffffffff81820754>] io_schedule_timeout+0xa4/0x110
[ 2817.726244] [<ffffffff81821a1b>] bit_wait_io+0x1b/0x70
[ 2817.727021] [<ffffffff818215ad>] __wait_on_bit+0x5d/0x90
[ 2817.727800] [<ffffffff8118f79d>] wait_on_page_bit_killable+0xcd/0xf0
[ 2817.728583] [<ffffffff810c3ab0>] ? autoremove_wake_function+0x40/0x40
[ 2817.729369] [<ffffffff8118f844>] __lock_page_or_retry+0x84/0xa0
[ 2817.730143] [<ffffffff8118fa4d>] filemap_fault+0x1ed/0x3f0
[ 2817.730920] [<ffffffff811bc1f0>] __do_fault+0x50/0xe0
[ 2817.731697] [<ffffffff811bfc2b>] handle_mm_fault+0xf8b/0x1820
[ 2817.732475] [<ffffffff8125575a>] ? ep_poll+0x21a/0x3d0
[ 2817.733252] [<ffffffff8106b537>] __do_page_fault+0x197/0x400
[ 2817.734035] [<ffffffff8106b7c2>] do_page_fault+0x22/0x30
[ 2817.734811] [<ffffffff81827478>] page_fault+0x28/0x30
[ 2817.906729] systemd-logind D ffff880157b6bb78 0 857 1 0x00000000
[ 2817.907551] ffff880157b6bb78 ffffffff8118d08e ffff880159470dc0 ffff880095402940
[ 2817.908386] ffff880157b6c000 ffff88015ec56d00 7fffffffffffffff ffffffff81821a00
[ 2817.909207] ffff880157b6bcd8 ffff880157b6bb90 ffffffff81821205 0000000000000000
[ 2817.910038] Call Trace:
[ 2817.910850] [<ffffffff8118d08e>] ? find_get_entry+0x1e/0xa0
[ 2817.911666] [<ffffffff81821a00>] ? bit_wait+0x60/0x60
[ 2817.912477] [<ffffffff81821205>] schedule+0x35/0x80
[ 2817.913289] [<ffffffff81824325>] schedule_timeout+0x1b5/0x270
[ 2817.914101] [<ffffffff811c008d>] ? handle_mm_fault+0x13ed/0x1820
[ 2817.914919] [<ffffffff81821a00>] ? bit_wait+0x60/0x60
[ 2817.915731] [<ffffffff81820754>] io_schedule_timeout+0xa4/0x110
[ 2817.916546] [<ffffffff81821a1b>] bit_wait_io+0x1b/0x70
[ 2817.917353] [<ffffffff818215ad>] __wait_on_bit+0x5d/0x90
[ 2817.918151] [<ffffffff8118f79d>] wait_on_page_bit_killable+0xcd/0xf0
[ 2817.918952] [<ffffffff810c3ab0>] ? autoremove_wake_function+0x40/0x40
[ 2817.919753] [<ffffffff8118f844>] __lock_page_or_retry+0x84/0xa0
[ 2817.920557] [<ffffffff8118fa4d>] filemap_fault+0x1ed/0x3f0
[ 2817.921349] [<ffffffff811bc1f0>] __do_fault+0x50/0xe0
[ 2817.922136] [<ffffffff811bfc2b>] handle_mm_fault+0xf8b/0x1820
[ 2817.922924] [<ffffffff8125575a>] ? ep_poll+0x21a/0x3d0
[ 2817.923714] [<ffffffff8106b537>] __do_page_fault+0x197/0x400
[ 2817.924503] [<ffffffff8106b7c2>] do_page_fault+0x22/0x30
[ 2817.925295] [<ffffffff81827478>] page_fault+0x28/0x30
[ 2818.096501] irqbalance D ffff880153413b78 0 1005 1 0x00000000
[ 2818.097266] ffff880153413b78 ffff880094347080 ffff880159472940 ffff880095406040
[ 2818.098039] ffff880153414000 ffff88015ecd6d00 7fffffffffffffff ffffffff81821a00
[ 2818.098814] ffff880153413cd8 ffff880153413b90 ffffffff81821205 0000000000000000
[ 2818.099591] Call Trace:
[ 2818.100346] [<ffffffff81821a00>] ? bit_wait+0x60/0x60
[ 2818.101114] [<ffffffff81821205>] schedule+0x35/0x80
[ 2818.101871] [<ffffffff81824325>] schedule_timeout+0x1b5/0x270
[ 2818.102626] [<ffffffff813b5cf3>] ? __blk_run_queue+0x33/0x40
[ 2818.103376] [<ffffffff813b5fba>] ? queue_unplugged+0x2a/0xb0
[ 2818.104147] [<ffffffff81821a00>] ? bit_wait+0x60/0x60
[ 2818.104894] [<ffffffff81820754>] io_schedule_timeout+0xa4/0x110
[ 2818.105638] [<ffffffff81821a1b>] bit_wait_io+0x1b/0x70
[ 2818.106392] [<ffffffff818215ad>] __wait_on_bit+0x5d/0x90
[ 2818.107142] [<ffffffff8118f79d>] wait_on_page_bit_killable+0xcd/0xf0
[ 2818.107919] [<ffffffff810c3ab0>] ? autoremove_wake_function+0x40/0x40
[ 2818.108674] [<ffffffff8118f844>] __lock_page_or_retry+0x84/0xa0
[ 2818.109435] [<ffffffff8118fa4d>] filemap_fault+0x1ed/0x3f0
[ 2818.110196] [<ffffffff811bc1f0>] __do_fault+0x50/0xe0
[ 2818.110956] [<ffffffff811bfc2b>] handle_mm_fault+0xf8b/0x1820
[ 2818.111729] [<ffffffff8106b537>] __do_page_fault+0x197/0x400
[ 2818.112489] [<ffffffff8106b7c2>] do_page_fault+0x22/0x30
[ 2818.113254] [<ffffffff81827478>] page_fault+0x28/0x30
9. avahi-daemon is blocked in pagefault handler with different
stacktrace
[ 2818.403217] avahi-daemon D ffff880090e671f8 0 3601 1 0x00000000
[ 2818.404027] ffff880090e671f8 ffffffff811eaac6 ffff8800350ce040 ffff880094910000
[ 2818.404832] ffff880090e68000 ffff880090e67230 ffff88015eccdd00 ffff88015eccdd00
[ 2818.405629] ffff880094910000 ffff880090e67210 ffffffff81821205 0000000100099ee5
[ 2818.406422] Call Trace:
[ 2818.407195] [<ffffffff811eaac6>] ? ___slab_alloc+0x2a6/0x460
[ 2818.407992] [<ffffffff81821205>] schedule+0x35/0x80
[ 2818.408775] [<ffffffff81824299>] schedule_timeout+0x129/0x270
[ 2818.409558] [<ffffffff810ec370>] ? trace_event_raw_event_tick_stop+0x120/0x120
[ 2818.410349] [<ffffffff810f574c>] ? ktime_get+0x3c/0xb0
[ 2818.411143] [<ffffffff81820754>] io_schedule_timeout+0xa4/0x110
[ 2818.411957] [<ffffffff811900c8>] mempool_alloc+0x148/0x170
[ 2818.412750] [<ffffffff810c3a70>] ? wake_atomic_t_function+0x60/0x60
[ 2818.413549] [<ffffffff813b20ad>] bio_alloc_bioset+0xbd/0x260
[ 2818.414342] [<ffffffff816a33cf>] __split_and_process_bio+0x1ef/0x3e0
[ 2818.415137] [<ffffffff816a3629>] dm_make_request+0x69/0xc0
[ 2818.415937] [<ffffffff813ba502>] generic_make_request+0xf2/0x1d0
[ 2818.416724] [<ffffffff813ba656>] submit_bio+0x76/0x170
[ 2818.417507] [<ffffffff811d2548>] __swap_writepage+0x228/0x270
[ 2818.418288] [<ffffffff811d7bcf>] ? __frontswap_store+0xdf/0x110
[ 2818.419067] [<ffffffff811d25c9>] swap_writepage+0x39/0x70
[ 2818.419857] [<ffffffff811a0b6d>] pageout.isra.44+0x16d/0x280
[ 2818.420639] [<ffffffff811a31ca>] shrink_page_list+0x3ca/0x7a0
[ 2818.421428] [<ffffffff811a3c39>] shrink_inactive_list+0x209/0x520
[ 2818.422212] [<ffffffff811a48d3>] shrink_lruvec+0x593/0x750
[ 2818.422997] [<ffffffff811a4b7f>] shrink_zone+0xef/0x2e0
[ 2818.423779] [<ffffffff811a4edc>] do_try_to_free_pages+0x16c/0x3f0
[ 2818.424558] [<ffffffff811a522e>] try_to_free_pages+0xce/0x180
[ 2818.425327] [<ffffffff81196eda>] __alloc_pages_nodemask+0x64a/0xb60
[ 2818.426100] [<ffffffff811e091c>] alloc_pages_current+0x8c/0x110
[ 2818.426869] [<ffffffff8118d77b>] __page_cache_alloc+0xab/0xc0
[ 2818.427652] [<ffffffff8119bf8b>] __do_page_cache_readahead+0xeb/0x230
[ 2818.428415] [<ffffffff8118d08e>] ? find_get_entry+0x1e/0xa0
[ 2818.429172] [<ffffffff8118fbd5>] filemap_fault+0x375/0x3f0
[ 2818.429932] [<ffffffff811bc1f0>] __do_fault+0x50/0xe0
[ 2818.430693] [<ffffffff811bfc2b>] handle_mm_fault+0xf8b/0x1820
[ 2818.431461] [<ffffffff81220f60>] ? poll_select_copy_remaining+0x140/0x140
[ 2818.432246] [<ffffffff8106b537>] __do_page_fault+0x197/0x400
[ 2818.433015] [<ffffffff8106b7c2>] do_page_fault+0x22/0x30
[ 2818.433782] [<ffffffff81827478>] page_fault+0x28/0x30
10. Memory hungry compilation job is blocked in pagefault handler with
similar trace to avahi
[ 2818.535321] cc1plus D ffff880090bcb388 0 3977 3976 0x00000000
[ 2818.536087] ffff880090bcb388 ffffffff811eaac6 ffffffff81e11500 ffff880094915280
[ 2818.536853] ffff880090bcc000 ffff880090bcb3c0 ffff88015ec0dd00 ffff88015ec0dd00
[ 2818.537618] ffff880094915280 ffff880090bcb3a0 ffffffff81821205 0000000100099ee5
[ 2818.538375] Call Trace:
[ 2818.539128] [<ffffffff811eaac6>] ? ___slab_alloc+0x2a6/0x460
[ 2818.539897] [<ffffffff81821205>] schedule+0x35/0x80
[ 2818.540665] [<ffffffff81824299>] schedule_timeout+0x129/0x270
[ 2818.541431] [<ffffffff810ec370>] ? trace_event_raw_event_tick_stop+0x120/0x120
[ 2818.542201] [<ffffffff810f574c>] ? ktime_get+0x3c/0xb0
[ 2818.542970] [<ffffffff81820754>] io_schedule_timeout+0xa4/0x110
[ 2818.543748] [<ffffffff811900c8>] mempool_alloc+0x148/0x170
[ 2818.544517] [<ffffffff810c3a70>] ? wake_atomic_t_function+0x60/0x60
[ 2818.545286] [<ffffffff813b20ad>] bio_alloc_bioset+0xbd/0x260
[ 2818.546063] [<ffffffff816a33cf>] __split_and_process_bio+0x1ef/0x3e0
[ 2818.546840] [<ffffffff816a3629>] dm_make_request+0x69/0xc0
[ 2818.547632] [<ffffffff813ba502>] generic_make_request+0xf2/0x1d0
[ 2818.548416] [<ffffffff813ba656>] submit_bio+0x76/0x170
[ 2818.549200] [<ffffffff811d2548>] __swap_writepage+0x228/0x270
[ 2818.549980] [<ffffffff811d7bcf>] ? __frontswap_store+0xdf/0x110
[ 2818.550763] [<ffffffff811d25c9>] swap_writepage+0x39/0x70
[ 2818.551546] [<ffffffff811a0b6d>] pageout.isra.44+0x16d/0x280
[ 2818.552341] [<ffffffff811a31ca>] shrink_page_list+0x3ca/0x7a0
[ 2818.553121] [<ffffffff811a3c39>] shrink_inactive_list+0x209/0x520
[ 2818.553902] [<ffffffff811a48d3>] shrink_lruvec+0x593/0x750
[ 2818.554677] [<ffffffff810986d1>] ? queue_work_on+0x31/0x40
[ 2818.555460] [<ffffffff810986d1>] ? queue_work_on+0x31/0x40
[ 2818.556230] [<ffffffff811a4b7f>] shrink_zone+0xef/0x2e0
[ 2818.557004] [<ffffffff811a4edc>] do_try_to_free_pages+0x16c/0x3f0
[ 2818.557784] [<ffffffff811a522e>] try_to_free_pages+0xce/0x180
[ 2818.558561] [<ffffffff81196eda>] __alloc_pages_nodemask+0x64a/0xb60
[ 2818.559337] [<ffffffff811e216e>] alloc_pages_vma+0xbe/0x240
[ 2818.560123] [<ffffffff811c012e>] handle_mm_fault+0x148e/0x1820
[ 2818.560884] [<ffffffff8106b537>] __do_page_fault+0x197/0x400
[ 2818.561643] [<ffffffff8106b7c2>] do_page_fault+0x22/0x30
[ 2818.562397] [<ffffffff81827478>] page_fault+0x28/0x30
11. disk events work is blocked in lru shrink
[ 2818.563150] kworker/u16:5 D ffff8800958135c8 0 3978 2 0x00000000
[ 2818.563922] Workqueue: events_freezable_power_ disk_events_workfn
[ 2818.564681] ffff8800958135c8 ffffffff811cc3a6 ffffffff81e11500 ffff8800350ce040
[ 2818.565456] ffff880095814000 ffff880095813600 ffff88015ec0dd00 ffff88015ec0dd00
[ 2818.566234] ffffffffffffffe0 ffff8800958135e0 ffffffff81821205 0000000100099b6b
[ 2818.567022] Call Trace:
[ 2818.567803] [<ffffffff811cc3a6>] ? page_referenced+0xb6/0x140
[ 2818.568593] [<ffffffff81821205>] schedule+0x35/0x80
[ 2818.569372] [<ffffffff81824299>] schedule_timeout+0x129/0x270
[ 2818.570152] [<ffffffff810ec370>] ? trace_event_raw_event_tick_stop+0x120/0x120
[ 2818.570936] [<ffffffff8182443e>] schedule_timeout_uninterruptible+0x1e/0x20
[ 2818.571720] [<ffffffff811aea13>] wait_iff_congested+0xf3/0x180
[ 2818.572505] [<ffffffff810c3a70>] ? wake_atomic_t_function+0x60/0x60
[ 2818.573283] [<ffffffff811a3f2f>] shrink_inactive_list+0x4ff/0x520
[ 2818.574056] [<ffffffff811a48d3>] shrink_lruvec+0x593/0x750
[ 2818.574829] [<ffffffff8120ebc0>] ? super_cache_count+0x70/0xe0
[ 2818.575610] [<ffffffff811a4b7f>] shrink_zone+0xef/0x2e0
[ 2818.576381] [<ffffffff811a4edc>] do_try_to_free_pages+0x16c/0x3f0
[ 2818.577158] [<ffffffff811a522e>] try_to_free_pages+0xce/0x180
[ 2818.577941] [<ffffffff81196eda>] __alloc_pages_nodemask+0x64a/0xb60
[ 2818.578717] [<ffffffff813b60b9>] ? alloc_request_struct+0x19/0x20
[ 2818.579488] [<ffffffff811e091c>] alloc_pages_current+0x8c/0x110
[ 2818.580262] [<ffffffff813b33e7>] bio_copy_kern+0xd7/0x1e0
[ 2818.581030] [<ffffffff813c0088>] blk_rq_map_kern+0xa8/0x140
[ 2818.581797] [<ffffffff813b9e5b>] ? blk_get_request+0x8b/0xf0
[ 2818.582558] [<ffffffff815adc68>] scsi_execute+0x78/0x1d0
[ 2818.583317] [<ffffffff815af96e>] scsi_execute_req_flags+0x8e/0xf0
[ 2818.584087] [<ffffffff815c0e27>] sr_check_events+0xb7/0x2d0
[ 2818.584845] [<ffffffff815fe0d8>] cdrom_check_events+0x18/0x30
[ 2818.585610] [<ffffffff815c125a>] sr_block_check_events+0x2a/0x30
[ 2818.586367] [<ffffffff813cb5f0>] disk_check_events+0x60/0x150
[ 2818.587119] [<ffffffff813cb6f6>] disk_events_workfn+0x16/0x20
[ 2818.587875] [<ffffffff8109a052>] process_one_work+0x162/0x480
[ 2818.588624] [<ffffffff8109a3bb>] worker_thread+0x4b/0x4c0
[ 2818.589353] [<ffffffff8109a370>] ? process_one_work+0x480/0x480
[ 2818.590090] [<ffffffff8109a370>] ? process_one_work+0x480/0x480
[ 2818.590817] [<ffffffff810a0588>] kthread+0xd8/0xf0
[ 2818.591535] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 2818.592274] [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[ 2818.592993] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
12. kernel workqueues have a lot of stuf waiting execution
[ 2819.710760]
[ 2819.711509] Showing busy workqueues and worker pools:
[ 2819.712278] workqueue events_freezable_power_: flags=0x86
[ 2819.713053] pwq 16: cpus=0-7 flags=0x4 nice=0 active=1/256
[ 2819.713843] in-flight: 3978:disk_events_workfn
[ 2819.714673] workqueue kcryptd: flags=0x2a
[ 2819.715464] pwq 16: cpus=0-7 flags=0x4 nice=0 active=4/4
[ 2819.716267] in-flight: 3966:kcryptd_crypt [dm_crypt], 160:kcryptd_crypt [dm_crypt], 3388:kcryptd_crypt [dm_crypt], 3339:kcryptd_crypt [dm_crypt]
[ 2819.717131] delayed: kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], <repeats enought times to make line 17983 characters wide>
[ 2822.339640] pool 16: cpus=0-7 flags=0x4 nice=0 workers=7 idle: 3979 3139
13. Swap usage stats from kernel:
[ 2960.187650] Swap cache stats: add 16096, delete 3484, find 193/223
[ 2960.187711] Free swap = 4054404kB
[ 2960.187748] Total swap = 4112380kB
Recovery from hang state:
Sometimes hang state can be recovred if I use sysrq-k and sysrq-e to kill all userspace programs. Systemd will then restart userspace process if hanged state was recovered successfully. Recovering doesn't work
Workaround:
Disabling swap with swapoff -a.
Running same compile job results to OOM killer cleanly killing the memory hungry process.
ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-4.4.0-22-generic 4.4.0-22.40
ProcVersionSignature: Ubuntu 4.4.0-22.40-generic 4.4.8
Uname: Linux 4.4.0-22-generic x86_64
NonfreeKernelModules: nvidia
ApportVersion: 2.20.1-0ubuntu2.1
Architecture: amd64
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path', '/dev/snd/hwC1D0', '/dev/snd/pcmC1D0c', '/dev/snd/pcmC1D0p', '/dev/snd/controlC1', '/dev/snd/hwC0D0', '/dev/snd/pcmC0D8p', '/dev/snd/pcmC0D7p', '/dev/snd/pcmC0D3p', '/dev/snd/controlC0', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CurrentDesktop: GNOME
Date: Mon Jun 6 17:55:26 2016
HibernationDevice: RESUME=UUID=04553608-429c-40dc-b53c-a200f2a70588
InstallationDate: Installed on 2037-12-25 (-7871 days ago)
InstallationMedia: Lubuntu 16.04 LTS "Xenial Xerus" - Release amd64 (20160420.1)
MachineType: Acer Aspire E5-571G
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-22-generic root=/dev/mapper/lubuntu--vg-root ro quiet splash
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
linux-restricted-modules-4.4.0-22-generic N/A
linux-backports-modules-4.4.0-22-generic N/A
linux-firmware 1.157
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 09/15/2015
dmi.bios.vendor: Insyde Corp.
dmi.bios.version: V1.32
dmi.board.name: EA50_HB
dmi.board.vendor: Acer
dmi.board.version: V1.32
dmi.chassis.type: 10
dmi.chassis.vendor: Acer
dmi.chassis.version: V1.32
dmi.modalias: dmi:bvnInsydeCorp.:bvrV1.32:bd09/15/2015:svnAcer:pnAspireE5-571G:pvrV1.32:rvnAcer:rnEA50_HB:rvrV1.32:cvnAcer:ct10:cvrV1.32:
dmi.product.name: Aspire E5-571G
dmi.product.version: V1.32
dmi.sys.vendor: Acer
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1589563/+subscriptions
Follow ups
