kernel-packages team mailing list archive

[Guy Baconniere <1495442@xxxxxxxxxxxxxxxxxx>]

apport information

** Attachment added: "ProcCpuinfo.txt"
   https://bugs.launchpad.net/bugs/1495442/+attachment/4463953/+files/ProcCpuinfo.txt

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1495442

Title:
  XFS: kernel hangs multiple times in xlog_cil_force_lsn

Status in linux package in Ubuntu:
  Incomplete

Bug description:
  Ubuntu 14.04.3 LTS with Kernel 3.19.0-26-generic

  One of our servers running rsnapshot and nfs-kernel-server on XFS partitions
  crashed or hanged two sundays in a raw.

  Kernel 3.19.0-26-generic hangs multiple times (same time) in xlog_cil_force_lsn (XFS).
  I assume this bug is not related with bug report 979498.

  I had also this issue with kernel 3.16.0-46-generic or 3.13.0-57-generic but I have
  no log during the crash to determine which of the above kernel was running.
  I switched to kernel 3.19.0-26-generic because 3.16.0-48-generic was not
  working well with Intel Gigabit -- tx hang (bug 1492146)

   INFO: task kthreadd:2 blocked for more than 120 seconds.
         Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
   "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
   kthreadd        D ffff8802362eb3a8     0     2      0 0x00000000
    ffff8802362eb3a8 ffff8802362c09d0 0000000000013e80 ffff8802362ebfd8
    0000000000013e80 ffff880235c28000 ffff8802362c09d0 ffff880200000001
    ffff8802362eb500 7fffffffffffffff ffff8802362eb4f8 ffff8802362c09d0
   Call Trace:
    [<ffffffff817b24f9>] schedule+0x29/0x70
    [<ffffffff817b55bc>] schedule_timeout+0x20c/0x280
    [<ffffffff810aa4a4>] ? update_curr+0xe4/0x180
    [<ffffffff817b3224>] wait_for_completion+0xa4/0x170
    [<ffffffff810a0a70>] ? wake_up_state+0x20/0x20
    [<ffffffff8108e7ed>] flush_work+0xed/0x1c0
    [<ffffffff8108ac40>] ? destroy_worker+0x90/0x90
    [<ffffffffc027451e>] xlog_cil_force_lsn+0x7e/0x1f0 [xfs]
    [<ffffffff8117d667>] ? __free_pages+0x27/0x30
    [<ffffffff811cd1a0>] ? __free_slab+0xd0/0x1f0
    [<ffffffffc0272c7e>] _xfs_log_force_lsn+0x5e/0x2d0 [xfs]
    [<ffffffffc0272f1e>] xfs_log_force_lsn+0x2e/0x90 [xfs]
    [<ffffffffc0264e79>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
    [<ffffffffc026198d>] __xfs_iunpin_wait+0x8d/0x120 [xfs]
    [<ffffffff810b4e50>] ? autoremove_wake_function+0x40/0x40
    [<ffffffffc0264e79>] xfs_iunpin_wait+0x19/0x20 [xfs]
    [<ffffffffc025a18c>] xfs_reclaim_inode+0x7c/0x340 [xfs]
    [<ffffffffc025a6a7>] xfs_reclaim_inodes_ag+0x257/0x370 [xfs]
    [<ffffffffc025b1e3>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
    [<ffffffffc0269f25>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
    [<ffffffff811efc79>] super_cache_scan+0x169/0x170
    [<ffffffff81188346>] shrink_node_slabs+0x1d6/0x370
    [<ffffffff8118af9a>] shrink_zone+0x20a/0x240
    [<ffffffff8118b335>] do_try_to_free_pages+0x155/0x440
    [<ffffffff8117b74f>] ? zone_watermark_ok+0x1f/0x30
    [<ffffffff8118b6da>] try_to_free_pages+0xba/0x150
    [<ffffffff8117f21b>] __alloc_pages_nodemask+0x61b/0xa60
    [<ffffffff8117f72a>] alloc_kmem_pages_node+0x6a/0x130
    [<ffffffff81072263>] ? copy_process.part.26+0xf3/0x1c00
    [<ffffffff81072283>] copy_process.part.26+0x113/0x1c00
    [<ffffffff810ab2ae>] ? dequeue_task_fair+0x44e/0x660
    [<ffffffff810ac001>] ? put_prev_entity+0x31/0x3f0
    [<ffffffff810af20c>] ? pick_next_task_fair+0x19c/0x880
    [<ffffffff81093730>] ? kthread_create_on_node+0x1c0/0x1c0
    [<ffffffff810ac3ef>] ? put_prev_task_fair+0x2f/0x50
    [<ffffffff81073f25>] do_fork+0xd5/0x340
    [<ffffffff810741b6>] kernel_thread+0x26/0x30
    [<ffffffff810941ea>] kthreadd+0x15a/0x1c0
    [<ffffffff81094090>] ? kthread_create_on_cpu+0x60/0x60
    [<ffffffff817b67d8>] ret_from_fork+0x58/0x90
    [<ffffffff81094090>] ? kthread_create_on_cpu+0x60/0x60

  10:17:01 xyz CRON[59020]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
  11:17:01 xyz CRON[59177]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
  12:00:01 xyz CRON[59282]: (root) CMD (/usr/bin/rsnapshot -c /etc/rsnapshot.conf hourly)
  12:03:05 xyz kernel: [616881.456247] INFO: task kthreadd:2 blocked for more than 120 seconds.
  12:03:05 xyz kernel: [616881.456587]       Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
  ...
  12:03:05 xyz kernel: [616881.457934] INFO: task kswapd0:57 blocked for more than 120 seconds.
  12:03:05 xyz kernel: [616881.458271]       Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
  ...
  12:03:05 xyz kernel: [616881.485629] INFO: task nfsd:2002 blocked for more than 120 seconds.
  12:03:05 xyz kernel: [616881.492761]       Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
  ...
  12:03:05 xyz kernel: [616881.523050] INFO: task snmpd:2155 blocked for more than 120 seconds.
  12:03:05 xyz kernel: [616881.530783]       Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
  ...
  12:03:05 xyz kernel: [616881.562789] INFO: task kworker/1:0:59030 blocked for more than 120 seconds.
  12:03:05 xyz kernel: [616881.571082]       Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
  ...
  12:03:05 xyz kernel: [616881.605747] INFO: task kworker/3:1H:59110 blocked for more than 120 seconds.
  12:03:05 xyz kernel: [616881.615112]       Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
  ...
  12:03:05 xyz kernel: [616881.652107] INFO: task rm:59284 blocked for more than 120 seconds.
  12:03:05 xyz kernel: [616881.661441]       Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
  ...
  12:05:05 xyz kernel: [617001.782228] INFO: task kswapd0:57 blocked for more than 120 seconds.
  12:05:05 xyz kernel: [617001.791580]       Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
  ...
  12:05:05 xyz kernel: [617001.828636] INFO: task irqbalance:1353 blocked for more than 120 seconds.
  12:05:05 xyz kernel: [617001.838019]       Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
  ...
  12:05:05 xyz kernel: [617001.875425] INFO: task nfsd:2002 blocked for more than 120 seconds.
  12:05:05 xyz kernel: [617001.884799]       Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
  ...

  REFERENCES
  http://marc.info/?l=linux-xfs&w=2&r=1&s=xlog_cil_force_lsn%2B0x7e%2F0x1f0&q=b
  http://marc.info/?l=linux-xfs&w=2&r=1&s=xlog_cil_force_lsn&q=b
  http://oss.sgi.com/cgi-bin/namazu.cgi?query=xlog_cil_force_lsn&submit=Search!&idxname=xfs&max=100&result=short&sort=date%3Alate
  http://oss.sgi.com/archives/xfs/2015-03/msg00174.html
  http://oss.sgi.com/archives/xfs/2014-09/msg00102.html
  http://marc.info/?t=133455915600002&r=1&w=2
  https://bugs.launchpad.net/ubuntu/+source/linux/+bug/979498
  --- 
  AlsaDevices:
   total 0
   crw-rw---- 1 root audio 116,  1 Sep 14 07:30 seq
   crw-rw---- 1 root audio 116, 33 Sep 14 07:30 timer
  AplayDevices: Error: [Errno 2] No such file or directory
  ApportVersion: 2.14.1-0ubuntu3.13
  Architecture: amd64
  ArecordDevices: Error: [Errno 2] No such file or directory
  AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
  CRDA: Error: [Errno 2] No such file or directory
  DistroRelease: Ubuntu 14.04
  HibernationDevice: RESUME=/dev/mapper/mainvg-swap
  IwConfig: Error: [Errno 2] No such file or directory
  MachineType: HP ProLiant DL380e Gen8
  Package: linux (not installed)
  PciMultimedia:
   
  ProcEnviron:
   LANGUAGE=en_US:en
   TERM=xterm
   PATH=(custom, no user)
   LANG=en_US.UTF-8
   SHELL=/bin/bash
  ProcFB: 0 VESA VGA
  ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.19.0-28-generic root=/dev/mapper/mainvg-root ro elevator=deadline nohz=off transparent_hugepage=always intel_iommu=on acpi_irq_nobalance cgroup_enable=memory swapaccount=1 nomdmonddf nomdmonisw
  ProcVersionSignature: Ubuntu 3.19.0-28.30~14.04.1-generic 3.19.8-ckt5
  RelatedPackageVersions:
   linux-restricted-modules-3.19.0-28-generic N/A
   linux-backports-modules-3.19.0-28-generic  N/A
   linux-firmware                             1.127.15
  RfKill: Error: [Errno 2] No such file or directory
  StagingDrivers: visorutil
  Tags:  trusty staging
  Uname: Linux 3.19.0-28-generic x86_64
  UpgradeStatus: No upgrade log present (probably fresh install)
  UserGroups:
   
  _MarkForUpload: True
  dmi.bios.date: 02/10/2014
  dmi.bios.vendor: HP
  dmi.bios.version: P73
  dmi.chassis.type: 23
  dmi.chassis.vendor: HP
  dmi.modalias: dmi:bvnHP:bvrP73:bd02/10/2014:svnHP:pnProLiantDL380eGen8:pvr:cvnHP:ct23:cvr:
  dmi.product.name: ProLiant DL380e Gen8
  dmi.sys.vendor: HP

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1495442/+subscriptions