kernel-packages team mailing list archive

[Jani Jaakkola <jj.lousa@xxxxxxxxx>]

Public bug reported:

Currently latest Ubuntu Linux kernel image has a a bug, probably a race
condition, which happens when there are multiple kerberos nfs4 mounts in
/etc/fstab. This does not happen on every boot, so to reproduce this you
probably need a few retries. This happens using the current Ubuntu Linux
kernel in 14.04:

# cat /proc/version_signature 
Ubuntu 3.13.0-35.62-generic 3.13.11.6

Apparently you need to have _multiple_ kerberos NFS4 mounts in
/etc/fstab  to trigger this:

xxxxxx.helsinki.fi:/root_vdm_3/fshome1/u1 /home/ad/fshome1/u1 nfs4 sec=krb5,rw,bg,hard 0 0
xxxxxx.helsinki.fi:/root_vdm_3/fshome2/u2 /home/ad/fshome2/u2 nfs4 sec=krb5,rw,bg,hard 0 0
xxxxxx.helsinki.fi:/root_vdm_3/fshome3/u3 /home/ad/fshome3/u3 nfs4 sec=krb5,rw,bg,hard 0 0
xxxxxx.helsinki.fi:/root_vdm_3/fshome4/u4 /home/ad/fshome4/u4 nfs4 sec=krb5,rw,bg,hard 0 0
xxxxxx.helsinki.fi:/root_vdm_3/fshome5/u5 /home/ad/fshome5/u5 nfs4 sec=krb5,rw,bg,hard 0 0
xxxxxx.helsinki.fi:/root_vdm_3/fshome6/u6 /home/ad/fshome6/u6 nfs4 sec=krb5,rw,bg,hard 0 0


 When this happens we get a kernel stack trace (complete trace included), which starts like this:


[   19.999751] gss_pipe_downcall: bad return from gss_fill_context: -4
[   19.999779] ------------[ cut here ]------------
[   19.999791] kernel BUG at /build/buildd/linux-3.13.0/net/sunrpc/auth_gss/auth_gss.c:735!
[   19.999796] invalid opcode: 0000 [#1] SMP 
[   19.999802] Modules linked in: arc4(+) des_generic cmac xcbc nfsv4 rmd160 crypto_null af_key xfrm_algo dm_crypt snd_hda_codec_realtek gpio_ich hp_wmi sparse_keymap snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_page_alloc snd_seq_midi snd_seq_midi_event snd_rawmidi coretemp dm_multipath snd_seq kvm_intel scsi_dh kvm snd_seq_device bnep serio_raw rfcomm snd_timer bluetooth lpc_ich snd soundcore tpm_infineon mei_me mei mac_hid parport_pc ppdev lp parport binfmt_misc rpcsec_gss_krb5 nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache dm_mirror dm_region_hash dm_log hid_generic usbhid hid nouveau mxm_wmi video i2c_algo_bit ttm e1000e drm_kms_helper ahci psmouse libahci drm wmi ptp pps_core

When this has happened, rpc.gssd gets stuck in D state:

# ps aux|grep gssd
root       452  0.0  0.0      0     0 ?        Ds   13:18   0:00 [rpc.gssd]

Also NFS4 mounts will fail, with an error message which does not tell
what is actually going on:

root@do0-kukad211-07:~# mount -a -t nfs4
mount.nfs4: access denied by server while mounting xxxx.helsinki.fi:/root_vdm_3/fshome1/u1
mount.nfs4: access denied by server while mounting xxxx.helsinki.fi:/root_vdm_3/fshome2/u2
mount.nfs4: access denied by server while mounting xxxx.helsinki.fi:/root_vdm_3/fshome3/u3
mount.nfs4: access denied by server while mounting xxxx.helsinki.fi:/root_vdm_3/fshome4/u4
mount.nfs4: access denied by server while mounting xxxx..helsinki.fi:/root_vdm_3/fshome5/u5
mount.nfs4: access denied by server while mounting xxxx.helsinki.fi:/root_vdm_3/fshome6/u6

This also happened on Ubuntu 12.04, so the bug is probably old. There is
a bug report, which is (IMHO) incorrectly reported against nfs-utils:
https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/1046762

We will fix this by removing the NFS mounts from fstab and doing them
sequentially in startup scripts, but it would be nice if the kernel race
would be fixed too.

** Affects: linux (Ubuntu)
     Importance: Undecided
         Status: New


** Tags: gssd kerberos nfs4

** Attachment added: "dmesg output"
   https://bugs.launchpad.net/bugs/1365485/+attachment/4195365/+files/dmesg.txt

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1365485

Title:
  Kernel BUG with multiple NFS4 kerberos mounts on boot

Status in “linux” package in Ubuntu:
  New

Bug description:
  Currently latest Ubuntu Linux kernel image has a a bug, probably a
  race condition, which happens when there are multiple kerberos nfs4
  mounts in /etc/fstab. This does not happen on every boot, so to
  reproduce this you probably need a few retries. This happens using the
  current Ubuntu Linux kernel in 14.04:

  # cat /proc/version_signature 
  Ubuntu 3.13.0-35.62-generic 3.13.11.6

  Apparently you need to have _multiple_ kerberos NFS4 mounts in
  /etc/fstab  to trigger this:

  xxxxxx.helsinki.fi:/root_vdm_3/fshome1/u1 /home/ad/fshome1/u1 nfs4 sec=krb5,rw,bg,hard 0 0
  xxxxxx.helsinki.fi:/root_vdm_3/fshome2/u2 /home/ad/fshome2/u2 nfs4 sec=krb5,rw,bg,hard 0 0
  xxxxxx.helsinki.fi:/root_vdm_3/fshome3/u3 /home/ad/fshome3/u3 nfs4 sec=krb5,rw,bg,hard 0 0
  xxxxxx.helsinki.fi:/root_vdm_3/fshome4/u4 /home/ad/fshome4/u4 nfs4 sec=krb5,rw,bg,hard 0 0
  xxxxxx.helsinki.fi:/root_vdm_3/fshome5/u5 /home/ad/fshome5/u5 nfs4 sec=krb5,rw,bg,hard 0 0
  xxxxxx.helsinki.fi:/root_vdm_3/fshome6/u6 /home/ad/fshome6/u6 nfs4 sec=krb5,rw,bg,hard 0 0

  
   When this happens we get a kernel stack trace (complete trace included), which starts like this:

  
  [   19.999751] gss_pipe_downcall: bad return from gss_fill_context: -4
  [   19.999779] ------------[ cut here ]------------
  [   19.999791] kernel BUG at /build/buildd/linux-3.13.0/net/sunrpc/auth_gss/auth_gss.c:735!
  [   19.999796] invalid opcode: 0000 [#1] SMP 
  [   19.999802] Modules linked in: arc4(+) des_generic cmac xcbc nfsv4 rmd160 crypto_null af_key xfrm_algo dm_crypt snd_hda_codec_realtek gpio_ich hp_wmi sparse_keymap snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_page_alloc snd_seq_midi snd_seq_midi_event snd_rawmidi coretemp dm_multipath snd_seq kvm_intel scsi_dh kvm snd_seq_device bnep serio_raw rfcomm snd_timer bluetooth lpc_ich snd soundcore tpm_infineon mei_me mei mac_hid parport_pc ppdev lp parport binfmt_misc rpcsec_gss_krb5 nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache dm_mirror dm_region_hash dm_log hid_generic usbhid hid nouveau mxm_wmi video i2c_algo_bit ttm e1000e drm_kms_helper ahci psmouse libahci drm wmi ptp pps_core

  When this has happened, rpc.gssd gets stuck in D state:

  # ps aux|grep gssd
  root       452  0.0  0.0      0     0 ?        Ds   13:18   0:00 [rpc.gssd]

  Also NFS4 mounts will fail, with an error message which does not tell
  what is actually going on:

  root@do0-kukad211-07:~# mount -a -t nfs4
  mount.nfs4: access denied by server while mounting xxxx.helsinki.fi:/root_vdm_3/fshome1/u1
  mount.nfs4: access denied by server while mounting xxxx.helsinki.fi:/root_vdm_3/fshome2/u2
  mount.nfs4: access denied by server while mounting xxxx.helsinki.fi:/root_vdm_3/fshome3/u3
  mount.nfs4: access denied by server while mounting xxxx.helsinki.fi:/root_vdm_3/fshome4/u4
  mount.nfs4: access denied by server while mounting xxxx..helsinki.fi:/root_vdm_3/fshome5/u5
  mount.nfs4: access denied by server while mounting xxxx.helsinki.fi:/root_vdm_3/fshome6/u6

  This also happened on Ubuntu 12.04, so the bug is probably old. There
  is a bug report, which is (IMHO) incorrectly reported against nfs-
  utils: https://bugs.launchpad.net/ubuntu/+source/nfs-
  utils/+bug/1046762

  We will fix this by removing the NFS mounts from fstab and doing them
  sequentially in startup scripts, but it would be nice if the kernel
  race would be fixed too.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1365485/+subscriptions