kernel-packages team mailing list archive

[Rafael David Tinoco <rafael.tinoco@xxxxxxxxxxxxx>]

For the kmem_cache_alloc problem I found a really promising fix saying
that the commit:

""""
commit ba5bb147330a8737b6b5a812cc774c79c070704b 
Author: Al Viro <viro@xxxxxxxxxxxxxxxxxx> 
Date: Thu Mar 21 02:21:19 2013 -0400 

pipe: take allocation and freeing of pipe_inode_info out of ->i_mutex 
""""

present from tags v3.10 to today....

caused a use-after-free problem on VERY rare occasions (maybe until your
workload was discovered). This problem really looks like the problem we
are having here and is fixed by the following commit:

""""
commit b0d8d2292160bb63de1972361ebed100c64b5b37 
Author: Linus Torvalds <torvalds@xxxxxxxxxxxxxxxxxxxx> 
Date: Mon Dec 2 09:44:51 2013 -0800 

vfs: fix subtle use-after-free of pipe_inode_info

The pipe code was trying (and failing) to be very careful about freeing 
the pipe info only after the last access, with a pattern like: 

spin_lock(&inode->i_lock); 
if (!--pipe->files) { 
inode->i_pipe = NULL; 
kill = 1; 
} 
spin_unlock(&inode->i_lock); 
__pipe_unlock(pipe); 
if (kill) 
free_pipe_info(pipe); 

where the final freeing is done last.

HOWEVER. The above is actually broken, because while the freeing is 
done at the end, if we have two racing processes releasing the pipe 
inode info, the one that *doesn't* free it will decrement the ->files 
count, and unlock the inode i_lock, but then still use the 
"pipe_inode_info" afterwards when it does the "__pipe_unlock(pipe)". 

This is *very* hard to trigger in practice, since the race window is 
very small, and adding debug options seems to just hide it by slowing 
things down. 

Simon originally reported this way back in July as an Oops in 
kmem_cache_allocate due to a single bit corruption (due to the final 
"spin_unlock(pipe->mutex.wait_lock)" incrementing a field in a different 
allocation that had re-used the free'd pipe-info), it's taken this long 
to figure out. 

Since the 'pipe->files' accesses aren't even protected by the pipe lock 
(we very much use the inode lock for that), the simple solution is to 
just drop the pipe lock early. And since there were two users of this 
pattern, create a helper function for it. 

Introduced commit ba5bb147330a ("pipe: take allocation and freeing of 
pipe_inode_info out of ->i_mutex"). 

Reported-by: Simon Kirby <sim@xxxxxxxxxx> 
Reported-by: Ian Applegate <ia@xxxxxxxxxxxxxx> 
Acked-by: Al Viro <viro@xxxxxxxxxxxxxxxxxx> 
Cc: stable@xxxxxxxxxx # v3.10+ 
Signed-off-by: Linus Torvalds <torvalds@xxxxxxxxxxxxxxxxxxxx> 
""""

The fix is contained in the following tags:

inaddy@inerddy:/bugs/kernel/upstream$ git tag --contains b0d8d2292160bb63de1972361ebed100c64b5b37 
v3.13 
v3.13-rc3 
v3.13-rc4 
v3.13-rc5 
v3.13-rc6 
v3.13-rc7 
v3.13-rc8 
v3.14 
v3.14-rc1 
v3.14-rc2 

And v3.13 might not suffer from this issue.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1389787

Title:
  3.11 memory consumption leads to HANG (not sure if 3.13 suffers from
  this).

Status in “linux” package in Ubuntu:
  Incomplete

Bug description:
  It was brought to my attention the following stack traces (occurring
  several times on different machines):

  crash> bt 
  PID: 2877 TASK: ffff881009b42ee0 CPU: 1 COMMAND: "git" 
  #0 [ffff880c0c6bb9d0] machine_kexec at ffffffff8104b141 
  #1 [ffff880c0c6bba40] crash_kexec at ffffffff810d5a58 
  #2 [ffff880c0c6bbb10] oops_end at ffffffff81748b38 
  #3 [ffff880c0c6bbb40] no_context at ffffffff8172dd02 
  #4 [ffff880c0c6bbb90] __bad_area_nosemaphore at ffffffff8172dee4 
  #5 [ffff880c0c6bbbf0] bad_area at ffffffff8172df5d 
  #6 [ffff880c0c6bbc20] __do_page_fault at ffffffff8174bda8 
  #7 [ffff880c0c6bbd30] do_page_fault at ffffffff8174bde7 
  #8 [ffff880c0c6bbd60] page_fault at ffffffff81747e98 
  [exception RIP: kmem_cache_alloc_trace+106] 
  RIP: ffffffff8119b22a RSP: ffff880c0c6bbe18 RFLAGS: 00010206 
  RAX: 0000000000000000 RBX: ffff88008407e0c0 RCX: 00000000031b6bc9 
  RDX: 00000000031b6bc8 RSI: 00000000000080d0 RDI: 00000000000173c0 
  RBP: ffff880c0c6bbe68 R8: ffff88081fa373c0 R9: 0000000000000000 
  R10: 0000000000000000 R11: 0000000000000202 R12: ffff88081f403800 
  R13: 0000000000629050 R14: ffffffff811bcfe4 R15: 00000000000080d0 
  ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 
  #9 [ffff880c0c6bbe70] alloc_pipe_info at ffffffff811bcfe4 
  #10 [ffff880c0c6bbe90] get_pipe_inode at ffffffff811bd0aa 
  #11 [ffff880c0c6bbeb0] create_pipe_files at ffffffff811bd648 
  #12 [ffff880c0c6bbef0] __do_pipe_flags at ffffffff811bd7d2 
  #13 [ffff880c0c6bbf30] sys_pipe2 at ffffffff811bd8f0 
  #14 [ffff880c0c6bbf70] sys_pipe at ffffffff811bd980 
  #15 [ffff880c0c6bbf80] system_call_fastpath at ffffffff8175099d 
  RIP: 00007f2d245da7e7 RSP: 00007fff6a4cfb18 RFLAGS: 00010283 
  RAX: 0000000000000016 RBX: ffffffff8175099d RCX: 000000000000001c 
  RDX: 00007f2d248aeac0 RSI: 0000000000000000 RDI: 00007fff6a4cfad0 
  RBP: 0000000000000002 R8: 0000000000000000 R9: 0000000000000000 
  R10: 0000000000000000 R11: 0000000000000202 R12: ffffffff811bd980 
  R13: ffff880c0c6bbf78 R14: 0000000000000000 R15: 000000000232efc0 
  ORIG_RAX: 0000000000000016 CS: 0033 SS: 002b

  AND

  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152703] SysRq : Show backtrace of all active CPUs 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152708] sending NMI to all CPUs: 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152714] NMI backtrace for cpu 0 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152720] CPU: 0 PID: 13579 Comm: python Tainted: GF I 3.11.0-15-generic #25~precise1-Ubuntu 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152721] Hardware name: HP ProLiant BL460c G7, BIOS I27 05/05/2011 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152724] task: ffff880085f50000 ti: ffff880a64afa000 task.ti: ffff880a64afa000 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152725] RIP: 0010:[<ffffffff81050ff5>] [<ffffffff81050ff5>] __ticket_spin_lock+0x25/0x30 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152736] RSP: 0018:ffff880a64afb748 EFLAGS: 00000293 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152737] RAX: 000000000000db25 RBX: ffff880be8576480 RCX: 000000018066002a 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152739] RDX: 000000000000db2e RSI: 0000000000000001 RDI: ffff880be8576480 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152740] RBP: ffff880a64afb748 R08: 0000000000000000 R09: ffffea002407db00 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152741] R10: ffffffff8128d8ed R11: 0000000000000001 R12: ffff8807572480b0 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152743] R13: ffff8807572481b8 R14: 00000000ffffffff R15: ffff8805e9c77908 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152745] FS: 00007ff00d4db700(0000) GS:ffff880c0ba00000(0000) knlGS:0000000000000000 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152746] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152748] CR2: 00007fb66906d8e0 CR3: 000000053b3dd000 CR4: 00000000000007f0 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152749] Stack: 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152750] ffff880a64afb758 ffffffff8174759e ffff880a64afb778 ffffffff8128eb02 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152756] ffff8807572480b0 ffff880757248200 ffff880a64afb798 ffffffff81270595 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152759] ffff880a64afb798 ffff8807572480b0 ffff880a64afb7c8 ffffffff81257621 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152763] Call Trace: 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152770] [<ffffffff8174759e>] _raw_spin_lock+0xe/0x20 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152776] [<ffffffff8128eb02>] ext4_es_lru_del+0x32/0x80 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152781] [<ffffffff81270595>] ext4_clear_inode+0x45/0x90 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152786] [<ffffffff81257621>] ext4_evict_inode+0x81/0x510 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152792] [<ffffffff811ce490>] evict+0xc0/0x1d0 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152795] [<ffffffff811ce5e1>] dispose_list+0x41/0x50 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152797] [<ffffffff8174756f>] ? _raw_spin_trylock+0xf/0x30 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152800] [<ffffffff811cf5d5>] prune_icache_sb+0x185/0x340 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152806] [<ffffffff811b7703>] prune_super+0x193/0x1b0 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152812] [<ffffffff8115af04>] shrink_slab+0x154/0x300 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152816] [<ffffffff8115dbf8>] do_try_to_free_pages+0x218/0x290 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152819] [<ffffffff8115dfa4>] try_to_free_pages+0xe4/0x1a0 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152823] [<ffffffff811520b8>] __alloc_pages_nodemask+0x618/0x9a0 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152827] [<ffffffff817336c8>] ? mem_cgroup_update_tree+0x9c/0x121 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152831] [<ffffffff811920a3>] alloc_pages_vma+0xa3/0x150 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152837] [<ffffffff81170249>] do_wp_page+0xc9/0x7c0 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152840] [<ffffffff81172d8c>] handle_pte_fault+0x1ec/0x230 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152842] [<ffffffff81173ff0>] handle_mm_fault+0x2a0/0x3e0 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152847] [<ffffffff8174b9ff>] __do_page_fault+0x1af/0x560 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152850] [<ffffffff811b8de7>] ? cp_new_stat+0x107/0x120 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152856] [<ffffffff810ca06c>] ? do_futex+0x7c/0x1b0 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152858] [<ffffffff811b91b5>] ? SYSC_newstat+0x25/0x30 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152861] [<ffffffff8174bde7>] do_page_fault+0x37/0x70 
  Oct 15 14:19:08 LGEARND8B5 kernel: [796493.152864] [<ffffffff81747e98>] page_fault+0x28/0x30

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1389787/+subscriptions