kernel-packages team mailing list archive

[Philipp Wendler <1566471@xxxxxxxxxxxxxxxxxx>]

I also experience this problem using the Xenial kernel 4.4.0-18.34~14.04.1 on Ubuntu 14.04.
I can even reproduce it as a non-root user by creating an overlay mount inside a user namespace.

After mounting an overlay over an NFS mount, I can successfully traverse
existing directories and create, write, read, and remove new files. As
soon as I try to read an existing file (from the lower layer NFS mount),
the application that attempts the read dies and the syslog shows the
kernel bug. The system continues running afterwards.

Furthermore, a similar crash occurs for NFS 4 mounts:

Apr 13 09:49:20 tortuga kernel: [ 4611.794037] BUG: unable to handle kernel NULL pointer dereference at 0000000000000160
Apr 13 09:49:20 tortuga kernel: [ 4611.794144] IP: [<ffffffffc088cd5d>] nfs4_file_open+0xcd/0x1d0 [nfsv4]
Apr 13 09:49:20 tortuga kernel: [ 4611.794202] PGD 414777067 PUD 302045067 PMD 0 
Apr 13 09:49:20 tortuga kernel: [ 4611.794233] Oops: 0000 [#1] SMP 
Apr 13 09:49:20 tortuga kernel: [ 4611.794255] Modules linked in: overlay rpcsec_gss_krb5 nfsv4 ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_CHECKSUM iptable_mangle xt_tcpudp ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables autofs4 bridge stp llc bnep rfcomm bluetooth nfsd auth_rpcgss nfs_acl nfs binfmt_misc lockd grace sunrpc fscache dm_crypt input_leds joydev snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi snd_hda_intel snd_hda_codec hid_generic snd_hda_core snd_hwdep intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp dcdbas snd_pcm kvm_intel snd_seq_midi snd_seq_midi_event kvm snd_rawmidi usbhid dm_multipath hid snd_seq snd_seq_device irqbypass crct10dif_pclmul snd_timer crc32_pclmul serio_raw snd aesni_intel mei_me aes_x86_64 soundcore lrw gf128mul mei glue_helper ablk_helper shpchp cryptd ppdev msr lpc_ich cpuid parport_pc 8250_fintek mac_hid lp parport amdkfd amd_iommu_v2 radeon i2c_algo_bit ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops e1000e drm ahci psmouse ptp libahci pps_core fjes video [last unloaded: ipmi_msghandler]
Apr 13 09:49:20 tortuga kernel: [ 4611.794983] CPU: 4 PID: 14306 Comm: cat Not tainted 4.4.0-18-generic #34~14.04.1-Ubuntu
Apr 13 09:49:20 tortuga kernel: [ 4611.795027] Hardware name: Dell Inc. OptiPlex 790/0HY9JP, BIOS A07 09/10/2011
Apr 13 09:49:20 tortuga kernel: [ 4611.795067] task: ffff8800a9822940 ti: ffff8803e9d30000 task.ti: ffff8803e9d30000
Apr 13 09:49:20 tortuga kernel: [ 4611.795108] RIP: 0010:[<ffffffffc088cd5d>]  [<ffffffffc088cd5d>] nfs4_file_open+0xcd/0x1d0 [nfsv4]
Apr 13 09:49:20 tortuga kernel: [ 4611.795171] RSP: 0018:ffff8803e9d33c18  EFLAGS: 00010246
Apr 13 09:49:20 tortuga kernel: [ 4611.795200] RAX: 0000000000000000 RBX: ffff8803e7d78700 RCX: ffff8803e9d33c38
Apr 13 09:49:20 tortuga kernel: [ 4611.795239] RDX: 0000000000008000 RSI: ffff8803f09a8540 RDI: ffff88041873a148
Apr 13 09:49:20 tortuga kernel: [ 4611.795278] RBP: ffff8803e9d33cb0 R08: 0000000000000000 R09: ffff88041cc03800
Apr 13 09:49:20 tortuga kernel: [ 4611.795317] R10: ffffffffc06c9230 R11: ffffea000f9f5e00 R12: 0000000000000000
Apr 13 09:49:20 tortuga kernel: [ 4611.795356] R13: ffff880317e9b680 R14: 0000000000000000 R15: ffff88041873a148
Apr 13 09:49:20 tortuga kernel: [ 4611.795396] FS:  00007f8678c77740(0000) GS:ffff88041d300000(0000) knlGS:0000000000000000
Apr 13 09:49:20 tortuga kernel: [ 4611.795440] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 13 09:49:20 tortuga kernel: [ 4611.795472] CR2: 0000000000000160 CR3: 0000000374f2b000 CR4: 00000000000406e0
Apr 13 09:49:20 tortuga kernel: [ 4611.795510] Stack:
Apr 13 09:49:20 tortuga kernel: [ 4611.795523]  ffff8803850868f0 ffffffff00008000 ffff880317d39740 ffff8803f09a8540
Apr 13 09:49:20 tortuga kernel: [ 4611.795568]  ffff880300008000 ffffffff00010000 ffff8803850868f0 0000000000000000
Apr 13 09:49:20 tortuga kernel: [ 4611.795612]  0000000000000000 ffff8803850868f0 ffff8803e7d78700 ffff8803e7d78710
Apr 13 09:49:20 tortuga kernel: [ 4611.795656] Call Trace:
Apr 13 09:49:20 tortuga kernel: [ 4611.795677]  [<ffffffff811fb397>] do_dentry_open+0x227/0x320
Apr 13 09:49:20 tortuga kernel: [ 4611.795720]  [<ffffffffc088cc90>] ? nfs4_file_fsync+0x180/0x180 [nfsv4]
Apr 13 09:49:20 tortuga kernel: [ 4611.795757]  [<ffffffff811fc467>] vfs_open+0x57/0x60
Apr 13 09:49:20 tortuga kernel: [ 4611.795787]  [<ffffffff8120ae8d>] path_openat+0x1ad/0x1310
Apr 13 09:49:20 tortuga kernel: [ 4611.795820]  [<ffffffff8120d05e>] do_filp_open+0x7e/0xd0
Apr 13 09:49:20 tortuga kernel: [ 4611.795852]  [<ffffffff812025bd>] ? cp_new_stat+0x13d/0x160
Apr 13 09:49:20 tortuga kernel: [ 4611.795885]  [<ffffffff8121a1e6>] ? __alloc_fd+0x46/0x180
Apr 13 09:49:20 tortuga kernel: [ 4611.795916]  [<ffffffff811fc7c9>] do_sys_open+0x129/0x270
Apr 13 09:49:20 tortuga kernel: [ 4611.795947]  [<ffffffff811fc92e>] SyS_open+0x1e/0x20
Apr 13 09:49:20 tortuga kernel: [ 4611.795978]  [<ffffffff817ee8f6>] entry_SYSCALL_64_fastpath+0x16/0x75
Apr 13 09:49:20 tortuga kernel: [ 4611.796013] Code: 00 00 49 8b 47 28 45 31 c0 48 8d 4d 88 8b 95 70 ff ff ff 48 8b 75 80 4c 89 ff 48 8b 80 58 04 00 00 48 8b 00 48 8b 80 e0 00 00 00 <ff> 90 60 01 00 00 48 3d 00 f0 ff ff 0f 87 ac 00 00 00 49 3b 45 
Apr 13 09:49:20 tortuga kernel: [ 4611.796194] RIP  [<ffffffffc088cd5d>] nfs4_file_open+0xcd/0x1d0 [nfsv4]
Apr 13 09:49:20 tortuga kernel: [ 4611.796242]  RSP <ffff8803e9d33c18>
Apr 13 09:49:20 tortuga kernel: [ 4611.796262] CR2: 0000000000000160
Apr 13 09:49:20 tortuga kernel: [ 4611.812656] ---[ end trace 7e26f22aae4f8eb6 ]---

I reproduced this crash also with the mainline 4.5 kernel.

I suspect that in both cases the actual bug is in overlayfs, and it might very well be the same bug, thus I am adding this here.
 If I should instead create a fresh bug, please tell me.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1566471

Title:
  kernel oops: NULL pointer dereference in
  nfs_inode_attach_open_context+0x37/0x70 [nfs]

Status in linux package in Ubuntu:
  Confirmed

Bug description:
  I'm attempting to boot a Xenial server install (created from
  debootstrap) via NFS with overlayroot so that the initial rootfs is
  read-only (via NFS) and all modifications are written to a tmpfs so
  that I can boot many such machines. The kernel oops occurs during run-
  init after the initramfs has successfully mounted the NFS rootfs,
  created the tmpfs, and the overlayfs using both. If I do not use
  overlayfs, and just boot into the NFS root (read-write), then
  everything works. Note that the following oops was gathered from a
  qemu virtual machine that I netbooted, though the apport output was
  from real hardware. The issue occurs in both cases. Please let me know
  if I can provide more information.

  + exec run-init /root /sbin/init
  [    9.003288] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
  [    9.005772] IP: [<ffffffffc01d14d7>] nfs_inode_attach_open_context+0x37/0x70 [nfs]
  [    9.007227] PGD 0 
  [    9.007227] Oops: 0002 [#1] SMP 
  [    9.007227] Modules linked in: overlay nfsv3 nfs_acl nfs lockd grace sunrpc fscache raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd psmouse floppy pata_acpi
  [    9.007227] CPU: 0 PID: 1 Comm: init Not tainted 4.4.0-16-generic #32-Ubuntu
  [    9.007227] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014
  [    9.007227] task: ffff88013ab80000 ti: ffff88013ab88000 task.ti: ffff88013ab88000
  [    9.007227] RIP: 0010:[<ffffffffc01d14d7>]  [<ffffffffc01d14d7>] nfs_inode_attach_open_context+0x37/0x70 [nfs]
  [    9.007227] RSP: 0018:ffff88013ab8bc30  EFLAGS: 00010246
  [    9.007227] RAX: ffff88007fa86d30 RBX: ffff8800bba16000 RCX: 0000000200000000
  [    9.007227] RDX: 0000000000000000 RSI: ffff88007fa86cc0 RDI: ffff8800bba16088
  [    9.007227] RBP: ffff88013ab8bc48 R08: ffff88007f09e09c R09: ffff88013b001800
  [    9.007227] R10: ffff88007fa86cc0 R11: 0000000000000000 R12: ffff88007fa86cc0
  [    9.007227] R13: ffff8800bba16088 R14: ffff8800bb9f7d88 R15: ffff88013a52f010
  [    9.007227] FS:  0000000000000000(0000) GS:ffff88013fc00000(0000) knlGS:0000000000000000
  [    9.007227] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  [    9.007227] CR2: 0000000000000008 CR3: 000000013a530000 CR4: 00000000001406f0
  [    9.007227] Stack:
  [    9.007227]  ffff88007fa86cc0 ffff88013a52f000 ffff8800bb9f7d88 ffff88013ab8bc58
  [    9.007227]  ffffffffc01d153b ffff88013ab8bc80 ffffffffc01d3d37 ffff88013a52f000
  [    9.007227]  ffff8800bb9f7d88 0000000000000000 ffff88013ab8bca0 ffffffffc01d010d
  [    9.007227] Call Trace:
  [    9.007227]  [<ffffffffc01d153b>] nfs_file_set_open_context+0x2b/0x30 [nfs]
  [    9.007227]  [<ffffffffc01d3d37>] nfs_open+0x37/0x60 [nfs]
  [    9.007227]  [<ffffffffc01d010d>] nfs_file_open+0x4d/0x70 [nfs]
  [    9.007227]  [<ffffffff812098cf>] do_dentry_open+0x1ff/0x310
  [    9.007227]  [<ffffffffc01d00c0>] ? nfs_file_fsync+0x130/0x130 [nfs]
  [    9.007227]  [<ffffffff8120aa76>] vfs_open+0x56/0x60
  [    9.007227]  [<ffffffff8121a107>] path_openat+0x1b7/0x1360
  [    9.007227]  [<ffffffff8121c4a1>] do_filp_open+0x91/0x100
  [    9.007227]  [<ffffffff81229da8>] ? __alloc_fd+0xc8/0x190
  [    9.007227]  [<ffffffff8120ae3e>] do_sys_open+0x13e/0x2a0
  [    9.007227]  [<ffffffff810a112d>] ? __put_cred+0x3d/0x50
  [    9.007227]  [<ffffffff8120a1f8>] ? SyS_access+0x1e8/0x230
  [    9.007227]  [<ffffffff8120afbe>] SyS_open+0x1e/0x20
  [    9.007227]  [<ffffffff81824ef2>] entry_SYSCALL_64_fastpath+0x16/0x71
  [    9.007227] Code: 54 53 48 8b 47 40 49 89 fc 48 8b 58 30 4c 8d ab 88 00 00 00 4c 89 ef e8 98 37 65 c1 48 8b 93 60 ff ff ff 49 8d 44 24 70 4c 89 ef <48> 89 42 08 49 89 54 24 70 48 8d 93 60 ff ff ff 49 89 54 24 78 
  [    9.007227] RIP  [<ffffffffc01d14d7>] nfs_inode_attach_open_context+0x37/0x70 [nfs]
  [    9.007227]  RSP <ffff88013ab8bc30>
  [    9.007227] CR2: 0000000000000008
  [    9.056135] ---[ end trace 4bf38e0df912649a ]---
  [    9.057055] BUG: unable to handle kernel NULL pointer dereference at 0000000000000158
  [    9.058345] IP: [<ffffffffc01d1c70>] __put_nfs_open_context+0xa0/0x100 [nfs]
  [    9.059479] PGD 0 
  [    9.059823] Oops: 0000 [#2] SMP 
  [    9.060117] Modules linked in: overlay nfsv3 nfs_acl nfs lockd grace sunrpc fscache raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd psmouse floppy pata_acpi
  [    9.060117] CPU: 0 PID: 1 Comm: init Tainted: G      D         4.4.0-16-generic #32-Ubuntu
  [    9.060117] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014
  [    9.060117] task: ffff88013ab80000 ti: ffff88013ab88000 task.ti: ffff88013ab88000
  [    9.060117] RIP: 0010:[<ffffffffc01d1c70>]  [<ffffffffc01d1c70>] __put_nfs_open_context+0xa0/0x100 [nfs]
  [    9.060117] RSP: 0018:ffff88013ab8b878  EFLAGS: 00010282
  [    9.060117] RAX: 0000000000000000 RBX: ffff880138e3e3c0 RCX: 0000000000000001
  [    9.060117] RDX: ffff88007fd3b358 RSI: 0000000000000001 RDI: ffff880138e3e3c0
  [    9.060117] RBP: ffff88013ab8b8a0 R08: 0000000000000000 R09: 0000000000000000
  [    9.060117] R10: ffff88007fd43598 R11: ffff8800bb71b610 R12: ffff88007fd3b3f8
  [    9.060117] R13: ffff88007fd3b480 R14: 0000000000000001 R15: ffff88007f09e000
  [    9.060117] FS:  0000000000000000(0000) GS:ffff88013fc00000(0000) knlGS:0000000000000000
  [    9.060117] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  [    9.060117] CR2: 0000000000000158 CR3: 0000000001e0a000 CR4: 00000000001406f0
  [    9.060117] Stack:
  [    9.060117]  ffff880138e3e3c0 ffff88007fd3b358 ffff88007fd3b480 ffff880138426620
  [    9.060117]  ffff88007fd38600 ffff88013ab8b8c8 ffffffffc01d3cf3 ffff8800bb71b600
  [    9.060117]  ffff88007fd43598 ffff88007fd43598 ffff88013ab8b8e8 ffffffffc01cfa8b
  [    9.060117] Call Trace:
  [    9.060117]  [<ffffffffc01d3cf3>] nfs_file_clear_open_context+0x83/0x90 [nfs]
  [    9.060117]  [<ffffffffc01cfa8b>] nfs_file_release+0x3b/0x50 [nfs]
  [    9.060117]  [<ffffffff8120db84>] __fput+0xe4/0x220
  [    9.060117]  [<ffffffff8120dcfe>] ____fput+0xe/0x10
  [    9.060117]  [<ffffffff8109d9e8>] task_work_run+0x78/0xa0
  [    9.060117]  [<ffffffff81082b64>] do_exit+0x2e4/0xae0
  [    9.060117]  [<ffffffff8101abf1>] oops_end+0xa1/0xd0
  [    9.060117]  [<ffffffff81069db5>] no_context+0x135/0x380
  [    9.060117]  [<ffffffff8106a080>] __bad_area_nosemaphore+0x80/0x1f0
  [    9.060117]  [<ffffffff8106a253>] bad_area+0x43/0x50
  [    9.060117]  [<ffffffff8106a76b>] __do_page_fault+0x35b/0x400
  [    9.060117]  [<ffffffff8106a877>] trace_do_page_fault+0x37/0xe0
  [    9.060117]  [<ffffffff81062f29>] do_async_page_fault+0x19/0x70
  [    9.060117]  [<ffffffff818270a8>] async_page_fault+0x28/0x30
  [    9.060117]  [<ffffffffc01d14d7>] ? nfs_inode_attach_open_context+0x37/0x70 [nfs]
  [    9.060117]  [<ffffffffc01d153b>] nfs_file_set_open_context+0x2b/0x30 [nfs]
  [    9.060117]  [<ffffffffc01d3d37>] nfs_open+0x37/0x60 [nfs]
  [    9.060117]  [<ffffffffc01d010d>] nfs_file_open+0x4d/0x70 [nfs]
  [    9.060117]  [<ffffffff812098cf>] do_dentry_open+0x1ff/0x310
  [    9.060117]  [<ffffffffc01d00c0>] ? nfs_file_fsync+0x130/0x130 [nfs]
  [    9.060117]  [<ffffffff8120aa76>] vfs_open+0x56/0x60
  [    9.060117]  [<ffffffff8121a107>] path_openat+0x1b7/0x1360
  [    9.060117]  [<ffffffff8121c4a1>] do_filp_open+0x91/0x100
  [    9.060117]  [<ffffffff81229da8>] ? __alloc_fd+0xc8/0x190
  [    9.060117]  [<ffffffff8120ae3e>] do_sys_open+0x13e/0x2a0
  [    9.060117]  [<ffffffff810a112d>] ? __put_cred+0x3d/0x50
  [    9.060117]  [<ffffffff8120a1f8>] ? SyS_access+0x1e8/0x230
  [    9.060117]  [<ffffffff8120afbe>] SyS_open+0x1e/0x20
  [    9.060117]  [<ffffffff81824ef2>] entry_SYSCALL_64_fastpath+0x16/0x71
  [    9.060117] Code: 89 43 78 ff 14 25 08 bf e2 81 4d 85 e4 74 22 49 8b 44 24 28 44 89 f6 48 89 df 48 8b 80 58 04 00 00 48 8b 00 48 8b 80 e0 00 00 00 <ff> 90 58 01 00 00 48 8b 7b 48 48 85 ff 74 05 e8 bc e5 f7 ff 48 
  [    9.060117] RIP  [<ffffffffc01d1c70>] __put_nfs_open_context+0xa0/0x100 [nfs]
  [    9.060117]  RSP <ffff88013ab8b878>
  [    9.060117] CR2: 0000000000000158
  [    9.060117] ---[ end trace 4bf38e0df912649b ]---
  [    9.060117] Fixing recursive fault but reboot is needed!

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1566471/+subscriptions