kernel-packages team mailing list archive

[Guy Baconniere <1495442@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Ubuntu 14.04.3 LTS with Kernel 3.19.0-26-generic

One of our servers running rsnapshot and nfs-kernel-server on XFS partitions
crashed or hanged two sundays in a raw.

Kernel 3.19.0-26-generic hangs multiple times (same time) in xlog_cil_force_lsn (XFS).
I assume this bug is not related with bug report 979498.

I had also this issue with kernel 3.16.0-46-generic or 3.13.0-57-generic but I have
no log during the crash to determine which of the above kernel was running.
I switched to kernel 3.19.0-26-generic because 3.16.0-48-generic was not
working well with Intel Gigabit -- tx hang (bug 1492146)

 INFO: task kthreadd:2 blocked for more than 120 seconds.
       Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
 kthreadd        D ffff8802362eb3a8     0     2      0 0x00000000
  ffff8802362eb3a8 ffff8802362c09d0 0000000000013e80 ffff8802362ebfd8
  0000000000013e80 ffff880235c28000 ffff8802362c09d0 ffff880200000001
  ffff8802362eb500 7fffffffffffffff ffff8802362eb4f8 ffff8802362c09d0
 Call Trace:
  [<ffffffff817b24f9>] schedule+0x29/0x70
  [<ffffffff817b55bc>] schedule_timeout+0x20c/0x280
  [<ffffffff810aa4a4>] ? update_curr+0xe4/0x180
  [<ffffffff817b3224>] wait_for_completion+0xa4/0x170
  [<ffffffff810a0a70>] ? wake_up_state+0x20/0x20
  [<ffffffff8108e7ed>] flush_work+0xed/0x1c0
  [<ffffffff8108ac40>] ? destroy_worker+0x90/0x90
  [<ffffffffc027451e>] xlog_cil_force_lsn+0x7e/0x1f0 [xfs]
  [<ffffffff8117d667>] ? __free_pages+0x27/0x30
  [<ffffffff811cd1a0>] ? __free_slab+0xd0/0x1f0
  [<ffffffffc0272c7e>] _xfs_log_force_lsn+0x5e/0x2d0 [xfs]
  [<ffffffffc0272f1e>] xfs_log_force_lsn+0x2e/0x90 [xfs]
  [<ffffffffc0264e79>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
  [<ffffffffc026198d>] __xfs_iunpin_wait+0x8d/0x120 [xfs]
  [<ffffffff810b4e50>] ? autoremove_wake_function+0x40/0x40
  [<ffffffffc0264e79>] xfs_iunpin_wait+0x19/0x20 [xfs]
  [<ffffffffc025a18c>] xfs_reclaim_inode+0x7c/0x340 [xfs]
  [<ffffffffc025a6a7>] xfs_reclaim_inodes_ag+0x257/0x370 [xfs]
  [<ffffffffc025b1e3>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
  [<ffffffffc0269f25>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
  [<ffffffff811efc79>] super_cache_scan+0x169/0x170
  [<ffffffff81188346>] shrink_node_slabs+0x1d6/0x370
  [<ffffffff8118af9a>] shrink_zone+0x20a/0x240
  [<ffffffff8118b335>] do_try_to_free_pages+0x155/0x440
  [<ffffffff8117b74f>] ? zone_watermark_ok+0x1f/0x30
  [<ffffffff8118b6da>] try_to_free_pages+0xba/0x150
  [<ffffffff8117f21b>] __alloc_pages_nodemask+0x61b/0xa60
  [<ffffffff8117f72a>] alloc_kmem_pages_node+0x6a/0x130
  [<ffffffff81072263>] ? copy_process.part.26+0xf3/0x1c00
  [<ffffffff81072283>] copy_process.part.26+0x113/0x1c00
  [<ffffffff810ab2ae>] ? dequeue_task_fair+0x44e/0x660
  [<ffffffff810ac001>] ? put_prev_entity+0x31/0x3f0
  [<ffffffff810af20c>] ? pick_next_task_fair+0x19c/0x880
  [<ffffffff81093730>] ? kthread_create_on_node+0x1c0/0x1c0
  [<ffffffff810ac3ef>] ? put_prev_task_fair+0x2f/0x50
  [<ffffffff81073f25>] do_fork+0xd5/0x340
  [<ffffffff810741b6>] kernel_thread+0x26/0x30
  [<ffffffff810941ea>] kthreadd+0x15a/0x1c0
  [<ffffffff81094090>] ? kthread_create_on_cpu+0x60/0x60
  [<ffffffff817b67d8>] ret_from_fork+0x58/0x90
  [<ffffffff81094090>] ? kthread_create_on_cpu+0x60/0x60

10:17:01 xyz CRON[59020]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
11:17:01 xyz CRON[59177]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
12:00:01 xyz CRON[59282]: (root) CMD (/usr/bin/rsnapshot -c /etc/rsnapshot.conf hourly)
12:03:05 xyz kernel: [616881.456247] INFO: task kthreadd:2 blocked for more than 120 seconds.
12:03:05 xyz kernel: [616881.456587]       Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
...
12:03:05 xyz kernel: [616881.457934] INFO: task kswapd0:57 blocked for more than 120 seconds.
12:03:05 xyz kernel: [616881.458271]       Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
...
12:03:05 xyz kernel: [616881.485629] INFO: task nfsd:2002 blocked for more than 120 seconds.
12:03:05 xyz kernel: [616881.492761]       Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
...
12:03:05 xyz kernel: [616881.523050] INFO: task snmpd:2155 blocked for more than 120 seconds.
12:03:05 xyz kernel: [616881.530783]       Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
...
12:03:05 xyz kernel: [616881.562789] INFO: task kworker/1:0:59030 blocked for more than 120 seconds.
12:03:05 xyz kernel: [616881.571082]       Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
...
12:03:05 xyz kernel: [616881.605747] INFO: task kworker/3:1H:59110 blocked for more than 120 seconds.
12:03:05 xyz kernel: [616881.615112]       Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
...
12:03:05 xyz kernel: [616881.652107] INFO: task rm:59284 blocked for more than 120 seconds.
12:03:05 xyz kernel: [616881.661441]       Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
...
12:05:05 xyz kernel: [617001.782228] INFO: task kswapd0:57 blocked for more than 120 seconds.
12:05:05 xyz kernel: [617001.791580]       Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
...
12:05:05 xyz kernel: [617001.828636] INFO: task irqbalance:1353 blocked for more than 120 seconds.
12:05:05 xyz kernel: [617001.838019]       Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
...
12:05:05 xyz kernel: [617001.875425] INFO: task nfsd:2002 blocked for more than 120 seconds.
12:05:05 xyz kernel: [617001.884799]       Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
...

REFERENCES
http://marc.info/?l=linux-xfs&w=2&r=1&s=xlog_cil_force_lsn%2B0x7e%2F0x1f0&q=b
http://marc.info/?l=linux-xfs&w=2&r=1&s=xlog_cil_force_lsn&q=b
http://oss.sgi.com/cgi-bin/namazu.cgi?query=xlog_cil_force_lsn&submit=Search!&idxname=xfs&max=100&result=short&sort=date%3Alate
http://oss.sgi.com/archives/xfs/2015-03/msg00174.html
http://oss.sgi.com/archives/xfs/2014-09/msg00102.html
http://marc.info/?t=133455915600002&r=1&w=2
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/979498

** Affects: linux (Ubuntu)
     Importance: Undecided
         Status: New

** Attachment added: "kern.log"
   https://bugs.launchpad.net/bugs/1495442/+attachment/4463892/+files/kern.log

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1495442

Title:
  XFS: kernel hangs multiple times in xlog_cil_force_lsn

Status in linux package in Ubuntu:
  New

Bug description:
  Ubuntu 14.04.3 LTS with Kernel 3.19.0-26-generic

  One of our servers running rsnapshot and nfs-kernel-server on XFS partitions
  crashed or hanged two sundays in a raw.

  Kernel 3.19.0-26-generic hangs multiple times (same time) in xlog_cil_force_lsn (XFS).
  I assume this bug is not related with bug report 979498.

  I had also this issue with kernel 3.16.0-46-generic or 3.13.0-57-generic but I have
  no log during the crash to determine which of the above kernel was running.
  I switched to kernel 3.19.0-26-generic because 3.16.0-48-generic was not
  working well with Intel Gigabit -- tx hang (bug 1492146)

   INFO: task kthreadd:2 blocked for more than 120 seconds.
         Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
   "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
   kthreadd        D ffff8802362eb3a8     0     2      0 0x00000000
    ffff8802362eb3a8 ffff8802362c09d0 0000000000013e80 ffff8802362ebfd8
    0000000000013e80 ffff880235c28000 ffff8802362c09d0 ffff880200000001
    ffff8802362eb500 7fffffffffffffff ffff8802362eb4f8 ffff8802362c09d0
   Call Trace:
    [<ffffffff817b24f9>] schedule+0x29/0x70
    [<ffffffff817b55bc>] schedule_timeout+0x20c/0x280
    [<ffffffff810aa4a4>] ? update_curr+0xe4/0x180
    [<ffffffff817b3224>] wait_for_completion+0xa4/0x170
    [<ffffffff810a0a70>] ? wake_up_state+0x20/0x20
    [<ffffffff8108e7ed>] flush_work+0xed/0x1c0
    [<ffffffff8108ac40>] ? destroy_worker+0x90/0x90
    [<ffffffffc027451e>] xlog_cil_force_lsn+0x7e/0x1f0 [xfs]
    [<ffffffff8117d667>] ? __free_pages+0x27/0x30
    [<ffffffff811cd1a0>] ? __free_slab+0xd0/0x1f0
    [<ffffffffc0272c7e>] _xfs_log_force_lsn+0x5e/0x2d0 [xfs]
    [<ffffffffc0272f1e>] xfs_log_force_lsn+0x2e/0x90 [xfs]
    [<ffffffffc0264e79>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
    [<ffffffffc026198d>] __xfs_iunpin_wait+0x8d/0x120 [xfs]
    [<ffffffff810b4e50>] ? autoremove_wake_function+0x40/0x40
    [<ffffffffc0264e79>] xfs_iunpin_wait+0x19/0x20 [xfs]
    [<ffffffffc025a18c>] xfs_reclaim_inode+0x7c/0x340 [xfs]
    [<ffffffffc025a6a7>] xfs_reclaim_inodes_ag+0x257/0x370 [xfs]
    [<ffffffffc025b1e3>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
    [<ffffffffc0269f25>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
    [<ffffffff811efc79>] super_cache_scan+0x169/0x170
    [<ffffffff81188346>] shrink_node_slabs+0x1d6/0x370
    [<ffffffff8118af9a>] shrink_zone+0x20a/0x240
    [<ffffffff8118b335>] do_try_to_free_pages+0x155/0x440
    [<ffffffff8117b74f>] ? zone_watermark_ok+0x1f/0x30
    [<ffffffff8118b6da>] try_to_free_pages+0xba/0x150
    [<ffffffff8117f21b>] __alloc_pages_nodemask+0x61b/0xa60
    [<ffffffff8117f72a>] alloc_kmem_pages_node+0x6a/0x130
    [<ffffffff81072263>] ? copy_process.part.26+0xf3/0x1c00
    [<ffffffff81072283>] copy_process.part.26+0x113/0x1c00
    [<ffffffff810ab2ae>] ? dequeue_task_fair+0x44e/0x660
    [<ffffffff810ac001>] ? put_prev_entity+0x31/0x3f0
    [<ffffffff810af20c>] ? pick_next_task_fair+0x19c/0x880
    [<ffffffff81093730>] ? kthread_create_on_node+0x1c0/0x1c0
    [<ffffffff810ac3ef>] ? put_prev_task_fair+0x2f/0x50
    [<ffffffff81073f25>] do_fork+0xd5/0x340
    [<ffffffff810741b6>] kernel_thread+0x26/0x30
    [<ffffffff810941ea>] kthreadd+0x15a/0x1c0
    [<ffffffff81094090>] ? kthread_create_on_cpu+0x60/0x60
    [<ffffffff817b67d8>] ret_from_fork+0x58/0x90
    [<ffffffff81094090>] ? kthread_create_on_cpu+0x60/0x60

  10:17:01 xyz CRON[59020]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
  11:17:01 xyz CRON[59177]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
  12:00:01 xyz CRON[59282]: (root) CMD (/usr/bin/rsnapshot -c /etc/rsnapshot.conf hourly)
  12:03:05 xyz kernel: [616881.456247] INFO: task kthreadd:2 blocked for more than 120 seconds.
  12:03:05 xyz kernel: [616881.456587]       Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
  ...
  12:03:05 xyz kernel: [616881.457934] INFO: task kswapd0:57 blocked for more than 120 seconds.
  12:03:05 xyz kernel: [616881.458271]       Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
  ...
  12:03:05 xyz kernel: [616881.485629] INFO: task nfsd:2002 blocked for more than 120 seconds.
  12:03:05 xyz kernel: [616881.492761]       Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
  ...
  12:03:05 xyz kernel: [616881.523050] INFO: task snmpd:2155 blocked for more than 120 seconds.
  12:03:05 xyz kernel: [616881.530783]       Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
  ...
  12:03:05 xyz kernel: [616881.562789] INFO: task kworker/1:0:59030 blocked for more than 120 seconds.
  12:03:05 xyz kernel: [616881.571082]       Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
  ...
  12:03:05 xyz kernel: [616881.605747] INFO: task kworker/3:1H:59110 blocked for more than 120 seconds.
  12:03:05 xyz kernel: [616881.615112]       Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
  ...
  12:03:05 xyz kernel: [616881.652107] INFO: task rm:59284 blocked for more than 120 seconds.
  12:03:05 xyz kernel: [616881.661441]       Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
  ...
  12:05:05 xyz kernel: [617001.782228] INFO: task kswapd0:57 blocked for more than 120 seconds.
  12:05:05 xyz kernel: [617001.791580]       Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
  ...
  12:05:05 xyz kernel: [617001.828636] INFO: task irqbalance:1353 blocked for more than 120 seconds.
  12:05:05 xyz kernel: [617001.838019]       Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
  ...
  12:05:05 xyz kernel: [617001.875425] INFO: task nfsd:2002 blocked for more than 120 seconds.
  12:05:05 xyz kernel: [617001.884799]       Tainted: G         C     3.19.0-26-generic #28~14.04.1-Ubuntu
  ...

  REFERENCES
  http://marc.info/?l=linux-xfs&w=2&r=1&s=xlog_cil_force_lsn%2B0x7e%2F0x1f0&q=b
  http://marc.info/?l=linux-xfs&w=2&r=1&s=xlog_cil_force_lsn&q=b
  http://oss.sgi.com/cgi-bin/namazu.cgi?query=xlog_cil_force_lsn&submit=Search!&idxname=xfs&max=100&result=short&sort=date%3Alate
  http://oss.sgi.com/archives/xfs/2015-03/msg00174.html
  http://oss.sgi.com/archives/xfs/2014-09/msg00102.html
  http://marc.info/?t=133455915600002&r=1&w=2
  https://bugs.launchpad.net/ubuntu/+source/linux/+bug/979498

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1495442/+subscriptions