← Back to team overview

kernel-packages team mailing list archive

[Bug 1505948] Re: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE)

 

Enabling KASAN on a Wily kernel prints the following:

Jan 27 12:02:05 ubuntu kernel: ==================================================================
Jan 27 12:02:05 ubuntu kernel: BUG: KASan: use after free in fuse_direct_IO+0xb1a/0xcc0 at addr ffff88036c414390
Jan 27 12:02:05 ubuntu kernel: Read of size 8 by task qemu-system-x86/2784
Jan 27 12:02:05 ubuntu kernel: =============================================================================
Jan 27 12:02:05 ubuntu kernel: BUG kmalloc-128 (Tainted: G I ): kasan: bad access detected
Jan 27 12:02:05 ubuntu kernel: -----------------------------------------------------------------------------
Jan 27 12:02:05 ubuntu kernel: Disabling lock debugging due to kernel taint
Jan 27 12:02:05 ubuntu kernel: INFO: Slab 0xffffea000db10500 objects=32 used=26 fp=0xffff88036c414e80 flags=0x2ffff0000000080
Jan 27 12:02:05 ubuntu kernel: INFO: Object 0xffff88036c414380 @offset=896 fp=0x (null)
Jan 27 12:02:05 ubuntu kernel: Bytes b4 ffff88036c414370: 18 00 00 00 40 27 a3 1f 3b 56 00 00 00 00 00 00 ....@'..;V......
Jan 27 12:02:05 ubuntu kernel: Object ffff88036c414380: 00 00 00 00 00 00 00 00 00 f0 75 35 00 00 00 00 ..........u5....
Jan 27 12:02:05 ubuntu kernel: Object ffff88036c414390: 80 27 67 81 ff ff ff ff 00 00 00 00 00 00 00 00 .'g.............
Jan 27 12:02:05 ubuntu kernel: Object ffff88036c4143a0: 05 00 00 00 00 00 00 00 80 82 44 ad 05 88 ff ff ..........D.....
Jan 27 12:02:05 ubuntu kernel: Object ffff88036c4143b0: 00 00 00 00 00 00 00 00 10 e1 bc 56 49 56 00 00 ...........VIV..
Jan 27 12:02:05 ubuntu kernel: Object ffff88036c4143c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Jan 27 12:02:05 ubuntu kernel: Object ffff88036c4143d0: 00 00 00 00 00 00 00 00 80 f6 85 6d 03 88 ff ff ...........m....
Jan 27 12:02:05 ubuntu kernel: Object ffff88036c4143e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Jan 27 12:02:05 ubuntu kernel: Object ffff88036c4143f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Jan 27 12:02:05 ubuntu kernel: CPU: 0 PID: 2784 Comm: qemu-system-x86 Tainted: G B I 4.2.0-25-generic 0000030
Jan 27 12:02:05 ubuntu kernel: Hardware name: IBM System x3550 M2 -[794654G]-/49Y6512 , BIOS -[D6E131CUS-1.05]- 11/25/2009
Jan 27 12:02:05 ubuntu kernel: ffff88036c414380 00000000d939cde9 ffff8805adf0f7c8 ffffffff828cafee
Jan 27 12:02:05 ubuntu kernel: 0000000000000080 ffff880373803680 ffff8805adf0f7f8 ffffffff81546759
Jan 27 12:02:05 ubuntu kernel: ffff880373803680 ffffea000db10500 ffff88036c414380 ffff8805ad56d600
Jan 27 12:02:05 ubuntu kernel: Call Trace:

Jan 27 12:02:05 ubuntu kernel: [< inline >] __dump_stack linux-4.2.0/lib/dump_stack.c:15
Jan 27 12:02:05 ubuntu kernel: [<ffffffff828cafee>] dump_stack+0x45/0x57 linux-4.2.0/lib/dump_stack.c:50
Jan 27 12:02:05 ubuntu kernel: [<ffffffff81546759>] print_trailer+0xf9/0x150 linux-4.2.0/mm/slub.c:650
Jan 27 12:02:05 ubuntu kernel: [<ffffffff8154b9c8>] object_err+0x38/0x50 linux-4.2.0/mm/slub.c:657
Jan 27 12:02:05 ubuntu kernel: [< inline >] print_address_description linux-4.2.0/mm/kasan/report.c:120
Jan 27 12:02:05 ubuntu kernel: [<ffffffff8154e3d8>] kasan_report_error+0x1e8/0x3f0 linux-4.2.0/mm/kasan/report.c:193
Jan 27 12:02:05 ubuntu kernel: [< inline >] kasan_report linux-4.2.0/mm/kasan/report.c:230
Jan 27 12:02:05 ubuntu kernel: [<ffffffff8154e791>] __asan_report_load8_noabort+0x61/0x70 linux-4.2.0/mm/kasan/report.c:251
Jan 27 12:02:05 ubuntu kernel: [<ffffffff818d8bfa>] fuse_direct_IO+0xb1a/0xcc0 linux-4.2.0/fs/fuse/file.c:2842
Jan 27 12:02:05 ubuntu kernel: [<ffffffff8145eda6>] generic_file_direct_write+0x246/0x540 linux-4.2.0/mm/filemap.c:2398
Jan 27 12:02:05 ubuntu kernel: [<ffffffff818da16c>] fuse_file_write_iter+0x31c/0x780 linux-4.2.0/fs/fuse/file.c:1182
Jan 27 12:02:05 ubuntu kernel: [<ffffffff81673aba>] aio_run_iocb+0x68a/0x870 linux-4.2.0/fs/aio.c:1446
Jan 27 12:02:05 ubuntu kernel: [< inline >] io_submit_one linux-4.2.0/fs/aio.c:1548
Jan 27 12:02:05 ubuntu kernel: [<ffffffff81676567>] do_io_submit+0x4a7/0xb40 linux-4.2.0/fs/aio.c:1606
Jan 27 12:02:05 ubuntu kernel: [< inline >] SYSC_io_submit linux-4.2.0/fs/aio.c:1631
Jan 27 12:02:05 ubuntu kernel: [<ffffffff81676c10>] SyS_io_submit+0x10/0x20 linux-4.2.0/fs/aio.c:1628
Jan 27 12:02:05 ubuntu kernel: [<ffffffff828dc632>] entry_SYSCALL_64_fastpath+0x16/0x75 linux-4.2.0/arch/x86/entry/entry_64.S:186
Jan 27 12:02:05 ubuntu kernel: Memory state around the buggy address:
Jan 27 12:02:05 ubuntu kernel: ffff88036c414280: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
Jan 27 12:02:05 ubuntu kernel: ffff88036c414300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Jan 27 12:02:05 ubuntu kernel: >ffff88036c414380: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
Jan 27 12:02:05 ubuntu kernel: ^
Jan 27 12:02:05 ubuntu kernel: ffff88036c414400: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Jan 27 12:02:05 ubuntu kernel: ffff88036c414480: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fc
Jan 27 12:02:05 ubuntu kernel: ==================================================================

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1505948

Title:
  Memory arena corruption with FUSE (was Memory allocation failure
  crashes kernel hard, presumably related to FUSE)

Status in linux package in Ubuntu:
  Confirmed
Status in linux source package in Wily:
  Confirmed

Bug description:
  Hello everybody,

  Linux 4.1, 4.2 or 4.3-rc leads to an immediate kernel panic in our
  setup when trying to start a Qemu process on top of a fuse-based
  mount. Here is an example stacktrace:

  [  739.807817] BUG: unable to handle kernel paging request at ffff8800a4104ea0
  [  739.840201] IP: [<ffffffff811cc95a>] kmem_cache_alloc_trace+0x7a/0x1f0
  [  739.870309] PGD 2fee067 PUD 2fbf4dd063 PMD 0
  [  739.890418] Oops: 0000 [#1] SMP
  [  739.905265] Modules linked in: nbd vport_vxlan vport_gre gre ebtable_filter ebtables openvswitch ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter xt_CT iptable_raw ip_tables xt_tcpudp ip6t_REJECT nf_reject_ipv6 xt_limit nf_conntrack_ipv6 nf_defrag_ipv6 xt_multiport xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables dm_crypt ipmi_ssif intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd kvm_intel kvm ipmi_devintf vhost_net vhost macvtap macvlan joydev input_leds dm_multipath scsi_dh bonding sb_edac 8021q garp hpilo mrp stp ipmi_si llc edac_core lpc_ich ioatdma 8250_fintek ipmi_msghandler lp shpchp acpi_power_meter mac_hid parport nls_iso8859_1 sch_fq_codel xfs libcrc32c btrfs xor raid6_pq ixgbe ses enclosure hid_generic dca vxlan usbhid ip6_udp_tunnel tg3 udp_tunnel ptp hid pps_core hpsa mdio wmi
  [  740.345300] CPU: 8 PID: 10550 Comm: qemu-system-x86 Not tainted 4.2.0-040200-generic #201508301530
  [  740.386879] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015
  [  740.416827] task: ffff882f8e958dc0 ti: ffff882f28c20000 task.ti: ffff882f28c20000
  [  740.451672] RIP: 0010:[<ffffffff811cc95a>]  [<ffffffff811cc95a>] kmem_cache_alloc_trace+0x7a/0x1f0
  [  740.494047] RSP: 0018:ffff882f28c23c68  EFLAGS: 00010286
  [  740.518425] RAX: 0000000000000000 RBX: 00000000000000d0 RCX: 00000000000026b3
  [  740.551611] RDX: 00000000000026b2 RSI: 00000000000000d0 RDI: ffff882fbf407840
  [  740.584846] RBP: ffff882f28c23ca8 R08: 0000000000019920 R09: ffffe8d000200ab0
  [  740.618287] R10: ffffffff812e8dcd R11: ffffea00bca0ac00 R12: 00000000000000d0
  [  740.651320] R13: ffff882fbf407840 R14: ffff8800a4104ea0 R15: ffff882fbf407840
  [  740.684195] FS:  00007f2642ffd700(0000) GS:ffff882fbfa00000(0000) knlGS:0000000000000000
  [  740.722030] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  [  740.749469] CR2: ffff8800a4104ea0 CR3: 0000002f26f83000 CR4: 00000000001426e0
  [  740.783390] Stack:
  [  740.792577]  ffffffff812e8dcd 0000000000000048 0000000000000002 ffff882f908c8468
  [  740.827003]  0000000001bef000 ffff882f928e4600 ffff882f28c23e48 ffff882f28c23d70
  [  740.860971]  ffff882f28c23d38 ffffffff812e8dcd 0000000000000001 ffff882f908c8300
  [  740.894994] Call Trace:
  [  740.906211]  [<ffffffff812e8dcd>] ? fuse_direct_IO+0xdd/0x280
  [  740.932940]  [<ffffffff812e8dcd>] fuse_direct_IO+0xdd/0x280
  [  740.958866]  [<ffffffff8117750e>] generic_file_direct_write+0x9e/0x150
  [  740.989318]  [<ffffffff812e96bc>] fuse_file_write_iter+0x15c/0x2e0
  [  741.017725]  [<ffffffff811e94a7>] __vfs_write+0xa7/0xf0
  [  741.041787]  [<ffffffff811e9b09>] vfs_write+0xa9/0x190
  [  741.065307]  [<ffffffff811ea9d9>] SyS_pwrite64+0x69/0xa0
  [  741.090141]  [<ffffffff81085b57>] ? SyS_rt_sigprocmask+0x67/0xb0
  [  741.135924]  [<ffffffff817a8e32>] entry_SYSCALL_64_fastpath+0x16/0x75
  [  741.183478] Code: 4c 03 05 32 d8 e3 7e 4d 8b 30 49 8b 40 10 4d 85 f6 0f 84 22 01 00 00 48 85 c0 0f 84 19 01 00 00 49 63 47 20 48 8d 4a 01 4d 8b 07 <49> 8b 1c 06 4c 89 f0 65 49 0f c7 08 0f 94 c0 84 c0 74 b9 49 63
  [  741.306817] RIP  [<ffffffff811cc95a>] kmem_cache_alloc_trace+0x7a/0x1f0

  The problem has also been documented by somebody else in the Fedora
  bug tracker at https://bugzilla.redhat.com/show_bug.cgi?id=1254310

  This behaviour is 100% reproducible. I have asked the fuse-devel
  mailinglist for advice, but up to this point with no success:

  http://sourceforge.net/p/fuse/mailman/message/34537139/

  We are still investigating if this issue is also happening with 4.0
  and will add the information to this bug report once we have it. Any
  help on debugging will be greatly appreciated.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505948/+subscriptions


References