← Back to team overview

group.of.nepali.translators team mailing list archive

[Bug 1781364] Re: Kernel error "task zfs:pid blocked for more than 120 seconds"

 

This bug was fixed in the package zfs-linux - 0.7.5-1ubuntu16.3

---------------
zfs-linux (0.7.5-1ubuntu16.3) bionic; urgency=medium

  * Fix zpl_mount() deadlock (LP: #1781364)
    - Upstream ZFS fix ac09630d8b0b ("Fix zpl_mount() deadlock")
      fixes deadlock on multiple parallelized mount/umounts

 -- Colin Ian King <colin.king@xxxxxxxxxxxxx>  Thu, 12 Jul 2018 09:18:24
+0100

** Changed in: zfs-linux (Ubuntu Bionic)
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of नेपाली
भाषा समायोजकहरुको समूह, which is subscribed to Xenial.
Matching subscriptions: Ubuntu 16.04 Bugs
https://bugs.launchpad.net/bugs/1781364

Title:
  Kernel error "task zfs:pid blocked for more than 120 seconds"

Status in Linux:
  Fix Released
Status in linux package in Ubuntu:
  Fix Released
Status in zfs-linux package in Ubuntu:
  Fix Released
Status in linux source package in Xenial:
  Fix Committed
Status in zfs-linux source package in Xenial:
  Confirmed
Status in linux source package in Bionic:
  Fix Committed
Status in zfs-linux source package in Bionic:
  Fix Released
Status in linux source package in Cosmic:
  Fix Released
Status in zfs-linux source package in Cosmic:
  Fix Released

Bug description:
  == SRU Justification, XENIAL, BIONIC ==

  Exercising ZFS with lxd with many mount/umounts can cause lockups and
  120 second timeout messages.

  == How to reproduce bug ==

  In a VM, 2 CPUs, 16GB of memory running Bionic:

  sudo apt update
  sudo apt install lxd lxd-client lxd-tools zfsutils-linux
  sudo lxd init

  (and with the default init options)

  then run:

  lxd-benchmark launch --count 96 --parallel 96

  This will reliably show the lockup every time without the fix.  With
  the fix (detailed below) one cannot reproduce the lockup.

  == Fix ==

  Upstream ZFS commit

  commit ac09630d8b0bf6c92084a30fdaefd03fd0adbdc1
  Author: Brian Behlendorf <behlendorf1@xxxxxxxx>
  Date: Wed Jul 11 15:49:10 2018 -0700

      Fix zpl_mount() deadlock

  == Regression Potential ==

  This just changes the locking in the mount path of ZFS and will only
  affect ZFS mount/unmounts.  The regression potential is small as this
  touches a very small code path that has been exhaustively exercises
  this code under multiple thread/CPU contention and shown not to break.

  ------------------

  ZFS bug report: https://github.com/zfsonlinux/zfs/issues/7691

  "I am using LXD containers that are configured to use a ZFS storage backend.
  I create many containers using a benchmark tool, which probably stresses the use of ZFS.
  In two out of four attempts, I got

  [  725.970508] INFO: task lxd:4455 blocked for more than 120 seconds.
  [  725.976730]       Tainted: P           O     4.15.0-20-generic #21-Ubuntu
  [  725.983551] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  [  725.991624] INFO: task txg_sync:4202 blocked for more than 120 seconds.
  [  725.998264]       Tainted: P           O     4.15.0-20-generic #21-Ubuntu
  [  726.005071] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  [  726.013313] INFO: task lxd:99919 blocked for more than 120 seconds.
  [  726.019609]       Tainted: P           O     4.15.0-20-generic #21-Ubuntu
  [  726.026418] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  [  726.034560] INFO: task zfs:100513 blocked for more than 120 seconds.
  [  726.040936]       Tainted: P           O     4.15.0-20-generic #21-Ubuntu
  [  726.047746] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  [  726.055791] INFO: task zfs:100584 blocked for more than 120 seconds.
  [  726.062170]       Tainted: P           O     4.15.0-20-generic #21-Ubuntu
  [  726.068979] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

  Describe how to reproduce the problem

      Start an Ubuntu 18.04 LTS server.
      Install LXD if not already installed.

  sudo apt update
  sudo apt install lxd lxd-client lxd-tools zfsutils-linux

      Configure LXD with sudo lxd init. When prompted for the storage
  backend, select ZFS and specify an empty disk.

  $ sudo lxd init
  Would you like to use LXD clustering? (yes/no) [default=no]:
   Do you want to configure a new storage pool? (yes/no) [default=yes]:
   Name of the new storage pool [default=default]:
   Name of the storage backend to use (dir, zfs) [default=zfs]:
   Create a new ZFS pool? (yes/no) [default=yes]:
   Would you like to use an existing block device? (yes/no) [default=no]: yes
   Path to the existing block device: /dev/sdb
   Would you like to connect to a MAAS server? (yes/no) [default=no]:
   Would you like to create a new local network bridge? (yes/no) [default=yes]: no
   Would you like to configure LXD to use an existing bridge or host interface? (yes/no) [default=no]: no
   Would you like LXD to be available over the network? (yes/no) [default=no]:
   Would you like stale cached images to be updated automatically? (yes/no) [default=yes]
   Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]:

      Now run the following to launch 48 containers in batches of 12.

  lxd-benchmark launch --count 48 --parallel 12

  In two out of four attempts, I got the kernel errors.

  I also tried

  echo 1 >/sys/module/spl/parameters/spl_taskq_kick

  but did not manage to continue.
  Include any warning/errors/backtraces from the system logs
  dmesg output

  [  725.970508] INFO: task lxd:4455 blocked for more than 120 seconds.
  [  725.976730]       Tainted: P           O     4.15.0-20-generic #21-Ubuntu
  [  725.983551] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  [  725.991408] lxd             D    0  4455      1 0x00000000
  [  725.991412] Call Trace:
  [  725.991424]  __schedule+0x297/0x8b0
  [  725.991428]  schedule+0x2c/0x80
  [  725.991429]  rwsem_down_write_failed+0x162/0x360
  [  725.991460]  ? dbuf_rele_and_unlock+0x1a8/0x4b0 [zfs]
  [  725.991465]  call_rwsem_down_write_failed+0x17/0x30
  [  725.991468]  ? call_rwsem_down_write_failed+0x17/0x30
  [  725.991469]  down_write+0x2d/0x40
  [  725.991472]  grab_super+0x30/0x90
  [  725.991501]  ? zpl_create+0x160/0x160 [zfs]
  [  725.991504]  sget_userns+0x91/0x490
  [  725.991507]  ? get_anon_bdev+0x100/0x100
  [  725.991534]  ? zpl_create+0x160/0x160 [zfs]
  [  725.991537]  sget+0x7d/0xa0
  [  725.991540]  ? get_anon_bdev+0x100/0x100
  [  725.991567]  zpl_mount+0xa8/0x160 [zfs]
  [  725.991570]  mount_fs+0x37/0x150
  [  725.991574]  vfs_kern_mount.part.23+0x5d/0x110
  [  725.991576]  do_mount+0x5ed/0xce0
  [  725.991577]  ? copy_mount_options+0x2c/0x220
  [  725.991578]  SyS_mount+0x98/0xe0
  [  725.991582]  do_syscall_64+0x73/0x130
  [  725.991583]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
  [  725.991585] RIP: 0033:0x4dbd5a
  [  725.991586] RSP: 002b:000000c428be6890 EFLAGS: 00000206 ORIG_RAX: 00000000000000a5
  [  725.991588] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00000000004dbd5a
  [  725.991589] RDX: 000000c421a04b7c RSI: 000000c426f94f40 RDI: 000000c4274ceaa0
  [  725.991590] RBP: 000000c428be6930 R08: 000000c425521a90 R09: 0000000000000000
  [  725.991590] R10: 0000000000000000 R11: 0000000000000206 R12: ffffffffffffffff
  [  725.991591] R13: 000000000000003e R14: 000000000000003d R15: 0000000000000080
  [  725.991624] INFO: task txg_sync:4202 blocked for more than 120 seconds.
  [  725.998264]       Tainted: P           O     4.15.0-20-generic #21-Ubuntu
  [  726.005071] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  [  726.012928] txg_sync        D    0  4202      2 0x80000000
  [  726.012930] Call Trace:
  [  726.012933]  __schedule+0x297/0x8b0
  [  726.012939]  schedule+0x2c/0x80
  [  726.012945]  cv_wait_common+0x11e/0x140 [spl]
  [  726.012948]  ? wait_woken+0x80/0x80
  [  726.012954]  __cv_wait+0x15/0x20 [spl]
  [  726.012981]  rrw_enter_write+0x3c/0xa0 [zfs]
  [  726.013006]  rrw_enter+0x13/0x20 [zfs]
  [  726.013033]  spa_sync+0x7c9/0xd80 [zfs]
  [  726.013062]  txg_sync_thread+0x2cd/0x4a0 [zfs]
  [  726.013089]  ? txg_quiesce_thread+0x3d0/0x3d0 [zfs]
  [  726.013093]  thread_generic_wrapper+0x74/0x90 [spl]
  [  726.013098]  kthread+0x121/0x140
  [  726.013101]  ? __thread_exit+0x20/0x20 [spl]
  [  726.013103]  ? kthread_create_worker_on_cpu+0x70/0x70
  [  726.013104]  ret_from_fork+0x22/0x40
  [  726.013313] INFO: task lxd:99919 blocked for more than 120 seconds.
  [  726.019609]       Tainted: P           O     4.15.0-20-generic #21-Ubuntu
  [  726.026418] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  [  726.034272] lxd             D    0 99919  99626 0x00000000
  [  726.034274] Call Trace:
  [  726.034277]  __schedule+0x297/0x8b0
  [  726.034283]  ? __wake_up_common+0x73/0x130
  [  726.034286]  schedule+0x2c/0x80
  [  726.034290]  cv_wait_common+0x11e/0x140 [spl]
  [  726.034293]  ? wait_woken+0x80/0x80
  [  726.034297]  __cv_wait+0x15/0x20 [spl]
  [  726.034322]  txg_wait_synced+0xdd/0x130 [zfs]
  [  726.034349]  zil_create+0x249/0x300 [zfs]
  [  726.034376]  zil_commit_writer+0x6d1/0x7c0 [zfs]
  [  726.034401]  zil_commit.part.14+0x93/0x100 [zfs]
  [  726.034429]  zil_commit+0x17/0x20 [zfs]
  [  726.034457]  zfs_sync+0x6e/0xb0 [zfs]
  [  726.034484]  zpl_sync_fs+0x50/0x80 [zfs]
  [  726.034490]  __sync_filesystem+0x33/0x60
  [  726.034493]  sync_filesystem+0x39/0x40
  [  726.034495]  generic_shutdown_super+0x27/0x120
  [  726.034496]  kill_anon_super+0x12/0x20
  [  726.034518]  zpl_kill_sb+0x1a/0x20 [zfs]
  [  726.034524]  deactivate_locked_super+0x48/0x80
  [  726.034529]  deactivate_super+0x40/0x60
  [  726.034532]  cleanup_mnt+0x3f/0x80
  [  726.034534]  __cleanup_mnt+0x12/0x20
  [  726.034535]  task_work_run+0x9d/0xc0
  [  726.034537]  exit_to_usermode_loop+0xc0/0xd0
  [  726.034539]  do_syscall_64+0x115/0x130
  [  726.034542]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
  [  726.034550] RIP: 0033:0x7fb553b3e8c7
  [  726.034551] RSP: 002b:00007fff426c5388 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6
  [  726.034553] RAX: 0000000000000000 RBX: 000000000000000f RCX: 00007fb553b3e8c7
  [  726.034553] RDX: 00007fb55476eb9f RSI: 0000000000000002 RDI: 00007fb554770b6b
  [  726.034554] RBP: 000000000000000c R08: 0000000000000000 R09: 00007fb553b8ae67
  [  726.034555] R10: 0000000000084000 R11: 0000000000000246 R12: 00007fff426c64a0
  [  726.034555] R13: 0000000003176690 R14: 0000000003177810 R15: 00000000031741f0
  [  726.034560] INFO: task zfs:100513 blocked for more than 120 seconds.
  [  726.040936]       Tainted: P           O     4.15.0-20-generic #21-Ubuntu
  [  726.047746] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  [  726.055600] zfs             D    0 100513   2626 0x00000000
  [  726.055602] Call Trace:
  [  726.055606]  __schedule+0x297/0x8b0
  [  726.055609]  schedule+0x2c/0x80
  [  726.055613]  cv_wait_common+0x11e/0x140 [spl]
  [  726.055615]  ? wait_woken+0x80/0x80
  [  726.055619]  __cv_wait+0x15/0x20 [spl]
  [  726.055642]  rrw_enter_read_impl+0x4e/0x160 [zfs]
  [  726.055666]  rrw_enter+0x1c/0x20 [zfs]
  [  726.055691]  dsl_pool_hold+0x5a/0x80 [zfs]
  [  726.055713]  dmu_objset_hold+0x33/0xa0 [zfs]
  [  726.055740]  zfs_ioc_objset_stats+0x32/0xa0 [zfs]
  [  726.055766]  zfsdev_ioctl+0x1e0/0x610 [zfs]
  [  726.055771]  do_vfs_ioctl+0xa8/0x630
  [  726.055774]  ? handle_mm_fault+0xb1/0x1f0
  [  726.055776]  ? __do_page_fault+0x270/0x4d0
  [  726.055777]  SyS_ioctl+0x79/0x90
  [  726.055779]  do_syscall_64+0x73/0x130
  [  726.055781]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
  [  726.055782] RIP: 0033:0x7fd4adc795d7
  [  726.055782] RSP: 002b:00007ffe356b8718 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
  [  726.055783] RAX: ffffffffffffffda RBX: 00007ffe356b8740 RCX: 00007fd4adc795d7
  [  726.055784] RDX: 00007ffe356b8740 RSI: 0000000000005a12 RDI: 0000000000000003
  [  726.055785] RBP: 000055632e278660 R08: 000000000000ffff R09: 00007fd4adcd1ed0
  [  726.055785] R10: 2f746c7561666564 R11: 0000000000000246 R12: 000055632e278660
  [  726.055788] R13: 00007ffe356beec0 R14: 00007fd4af1756e0 R15: 00007ffe356bbe30
  [  726.055791] INFO: task zfs:100584 blocked for more than 120 seconds.
  [  726.062170]       Tainted: P           O     4.15.0-20-generic #21-Ubuntu
  [  726.068979] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  [  726.076835] zfs             D    0 100584  15513 0x00000000
  [  726.076837] Call Trace:
  [  726.076840]  __schedule+0x297/0x8b0
  [  726.076845]  schedule+0x2c/0x80
  [  726.076851]  cv_wait_common+0x11e/0x140 [spl]
  [  726.076854]  ? wait_woken+0x80/0x80
  [  726.076859]  __cv_wait+0x15/0x20 [spl]
  [  726.076881]  rrw_enter_read_impl+0x4e/0x160 [zfs]
  [  726.076905]  rrw_enter+0x1c/0x20 [zfs]
  [  726.076929]  dsl_pool_hold+0x5a/0x80 [zfs]
  [  726.076951]  dmu_objset_hold+0x33/0xa0 [zfs]
  [  726.076977]  zfs_ioc_objset_stats+0x32/0xa0 [zfs]
  [  726.077001]  zfsdev_ioctl+0x1e0/0x610 [zfs]
  [  726.077005]  do_vfs_ioctl+0xa8/0x630
  [  726.077006]  ? handle_mm_fault+0xb1/0x1f0
  [  726.077008]  ? __do_page_fault+0x270/0x4d0
  [  726.077010]  SyS_ioctl+0x79/0x90
  [  726.077011]  do_syscall_64+0x73/0x130
  [  726.077013]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
  [  726.077014] RIP: 0033:0x7fc2734075d7
  [  726.077014] RSP: 002b:00007fff653a4b08 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
  [  726.077015] RAX: ffffffffffffffda RBX: 00007fff653a4b30 RCX: 00007fc2734075d7
  [  726.077016] RDX: 00007fff653a4b30 RSI: 0000000000005a12 RDI: 0000000000000003
  [  726.077017] RBP: 000055f3576e9660 R08: 000000000000ffff R09: 00007fc27345fed0
  [  726.077017] R10: 2f746c7561666564 R11: 0000000000000246 R12: 000055f3576e9660
  [  726.077018] R13: 00007fff653aaec0 R14: 00007fc2749036e0 R15: 00007fff653a8220
  [  846.801124] INFO: task lxd:4455 blocked for more than 120 seconds.
  [  846.807352]       Tainted: P           O     4.15.0-20-generic #21-Ubuntu
  [  846.814170] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  [  846.822028] lxd             D    0  4455      1 0x00000000
  [  846.822031] Call Trace:
  [  846.822042]  __schedule+0x297/0x8b0
  [  846.822045]  schedule+0x2c/0x80
  [  846.822047]  rwsem_down_write_failed+0x162/0x360
  [  846.822078]  ? dbuf_rele_and_unlock+0x1a8/0x4b0 [zfs]
  [  846.822083]  call_rwsem_down_write_failed+0x17/0x30
  [  846.822086]  ? call_rwsem_down_write_failed+0x17/0x30
  [  846.822087]  down_write+0x2d/0x40
  [  846.822091]  grab_super+0x30/0x90
  [  846.822118]  ? zpl_create+0x160/0x160 [zfs]
  [  846.822121]  sget_userns+0x91/0x490
  [  846.822123]  ? get_anon_bdev+0x100/0x100
  [  846.822150]  ? zpl_create+0x160/0x160 [zfs]
  [  846.822153]  sget+0x7d/0xa0
  [  846.822156]  ? get_anon_bdev+0x100/0x100
  [  846.822181]  zpl_mount+0xa8/0x160 [zfs]
  [  846.822183]  mount_fs+0x37/0x150
  [  846.822188]  vfs_kern_mount.part.23+0x5d/0x110
  [  846.822189]  do_mount+0x5ed/0xce0
  [  846.822190]  ? copy_mount_options+0x2c/0x220
  [  846.822192]  SyS_mount+0x98/0xe0
  [  846.822195]  do_syscall_64+0x73/0x130
  [  846.822196]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
  [  846.822198] RIP: 0033:0x4dbd5a
  [  846.822199] RSP: 002b:000000c428be6890 EFLAGS: 00000206 ORIG_RAX: 00000000000000a5
  [  846.822201] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00000000004dbd5a
  [  846.822203] RDX: 000000c421a04b7c RSI: 000000c426f94f40 RDI: 000000c4274ceaa0
  [  846.822205] RBP: 000000c428be6930 R08: 000000c425521a90 R09: 0000000000000000
  [  846.822206] R10: 0000000000000000 R11: 0000000000000206 R12: ffffffffffffffff
  [  846.822206] R13: 000000000000003e R14: 000000000000003d R15: 0000000000000080
  [  846.822239] INFO: task txg_sync:4202 blocked for more than 120 seconds.
  [  846.828882]       Tainted: P           O     4.15.0-20-generic #21-Ubuntu
  [  846.835692] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  [  846.843549] txg_sync        D    0  4202      2 0x80000000
  [  846.843551] Call Trace:
  [  846.843554]  __schedule+0x297/0x8b0
  [  846.843560]  schedule+0x2c/0x80
  [  846.843566]  cv_wait_common+0x11e/0x140 [spl]
  [  846.843570]  ? wait_woken+0x80/0x80
  [  846.843574]  __cv_wait+0x15/0x20 [spl]
  [  846.843603]  rrw_enter_write+0x3c/0xa0 [zfs]
  [  846.843629]  rrw_enter+0x13/0x20 [zfs]
  [  846.843654]  spa_sync+0x7c9/0xd80 [zfs]
  [  846.843682]  txg_sync_thread+0x2cd/0x4a0 [zfs]
  [  846.843708]  ? txg_quiesce_thread+0x3d0/0x3d0 [zfs]
  [  846.843713]  thread_generic_wrapper+0x74/0x90 [spl]
  [  846.843717]  kthread+0x121/0x140
  [  846.843720]  ? __thread_exit+0x20/0x20 [spl]
  [  846.843721]  ? kthread_create_worker_on_cpu+0x70/0x70
  [  846.843723]  ret_from_fork+0x22/0x40
  [  846.843931] INFO: task lxd:99919 blocked for more than 120 seconds.
  [  846.850227]       Tainted: P           O     4.15.0-20-generic #21-Ubuntu
  [  846.857040] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  [  846.864892] lxd             D    0 99919  99626 0x00000000
  [  846.864894] Call Trace:
  [  846.864897]  __schedule+0x297/0x8b0
  [  846.864903]  ? __wake_up_common+0x73/0x130
  [  846.864906]  schedule+0x2c/0x80
  [  846.864910]  cv_wait_common+0x11e/0x140 [spl]
  [  846.864912]  ? wait_woken+0x80/0x80
  [  846.864917]  __cv_wait+0x15/0x20 [spl]
  [  846.864942]  txg_wait_synced+0xdd/0x130 [zfs]
  [  846.864971]  zil_create+0x249/0x300 [zfs]
  [  846.864998]  zil_commit_writer+0x6d1/0x7c0 [zfs]
  [  846.865023]  zil_commit.part.14+0x93/0x100 [zfs]
  [  846.865051]  zil_commit+0x17/0x20 [zfs]
  [  846.865080]  zfs_sync+0x6e/0xb0 [zfs]
  [  846.865107]  zpl_sync_fs+0x50/0x80 [zfs]
  [  846.865111]  __sync_filesystem+0x33/0x60
  [  846.865113]  sync_filesystem+0x39/0x40
  [  846.865114]  generic_shutdown_super+0x27/0x120
  [  846.865116]  kill_anon_super+0x12/0x20
  [  846.865138]  zpl_kill_sb+0x1a/0x20 [zfs]
  [  846.865140]  deactivate_locked_super+0x48/0x80
  [  846.865143]  deactivate_super+0x40/0x60
  [  846.865145]  cleanup_mnt+0x3f/0x80
  [  846.865147]  __cleanup_mnt+0x12/0x20
  [  846.865148]  task_work_run+0x9d/0xc0
  [  846.865150]  exit_to_usermode_loop+0xc0/0xd0
  [  846.865152]  do_syscall_64+0x115/0x130
  [  846.865153]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
  [  846.865154] RIP: 0033:0x7fb553b3e8c7
  [  846.865155] RSP: 002b:00007fff426c5388 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6
  [  846.865156] RAX: 0000000000000000 RBX: 000000000000000f RCX: 00007fb553b3e8c7
  [  846.865158] RDX: 00007fb55476eb9f RSI: 0000000000000002 RDI: 00007fb554770b6b
  [  846.865159] RBP: 000000000000000c R08: 0000000000000000 R09: 00007fb553b8ae67
  [  846.865162] R10: 0000000000084000 R11: 0000000000000246 R12: 00007fff426c64a0
  [  846.865162] R13: 0000000003176690 R14: 0000000003177810 R15: 00000000031741f0
  [  846.865167] INFO: task zfs:100513 blocked for more than 120 seconds.
  [  846.871546]       Tainted: P           O     4.15.0-20-generic #21-Ubuntu
  [  846.878357] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  [  846.886214] zfs             D    0 100513   2626 0x00000000
  [  846.886215] Call Trace:
  [  846.886218]  __schedule+0x297/0x8b0
  [  846.886223]  schedule+0x2c/0x80
  [  846.886230]  cv_wait_common+0x11e/0x140 [spl]
  [  846.886232]  ? wait_woken+0x80/0x80
  [  846.886237]  __cv_wait+0x15/0x20 [spl]
  [  846.886261]  rrw_enter_read_impl+0x4e/0x160 [zfs]
  [  846.886284]  rrw_enter+0x1c/0x20 [zfs]
  [  846.886309]  dsl_pool_hold+0x5a/0x80 [zfs]
  [  846.886331]  dmu_objset_hold+0x33/0xa0 [zfs]
  [  846.886357]  zfs_ioc_objset_stats+0x32/0xa0 [zfs]
  [  846.886383]  zfsdev_ioctl+0x1e0/0x610 [zfs]
  [  846.886388]  do_vfs_ioctl+0xa8/0x630
  [  846.886391]  ? handle_mm_fault+0xb1/0x1f0
  [  846.886394]  ? __do_page_fault+0x270/0x4d0
  [  846.886396]  SyS_ioctl+0x79/0x90
  [  846.886397]  do_syscall_64+0x73/0x130
  [  846.886399]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
  [  846.886400] RIP: 0033:0x7fd4adc795d7
  [  846.886402] RSP: 002b:00007ffe356b8718 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
  [  846.886405] RAX: ffffffffffffffda RBX: 00007ffe356b8740 RCX: 00007fd4adc795d7
  [  846.886406] RDX: 00007ffe356b8740 RSI: 0000000000005a12 RDI: 0000000000000003
  [  846.886406] RBP: 000055632e278660 R08: 000000000000ffff R09: 00007fd4adcd1ed0
  [  846.886407] R10: 2f746c7561666564 R11: 0000000000000246 R12: 000055632e278660
  [  846.886408] R13: 00007ffe356beec0 R14: 00007fd4af1756e0 R15: 00007ffe356bbe30
  [  846.886410] INFO: task zfs:100584 blocked for more than 120 seconds.
  [  846.892790]       Tainted: P           O     4.15.0-20-generic #21-Ubuntu
  [  846.899598] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  [  846.907454] zfs             D    0 100584  15513 0x00000000
  [  846.907456] Call Trace:
  [  846.907459]  __schedule+0x297/0x8b0
  [  846.907465]  schedule+0x2c/0x80
  [  846.907470]  cv_wait_common+0x11e/0x140 [spl]
  [  846.907471]  ? wait_woken+0x80/0x80
  [  846.907477]  __cv_wait+0x15/0x20 [spl]
  [  846.907499]  rrw_enter_read_impl+0x4e/0x160 [zfs]
  [  846.907523]  rrw_enter+0x1c/0x20 [zfs]
  [  846.907547]  dsl_pool_hold+0x5a/0x80 [zfs]
  [  846.907568]  dmu_objset_hold+0x33/0xa0 [zfs]
  [  846.907594]  zfs_ioc_objset_stats+0x32/0xa0 [zfs]
  [  846.907618]  zfsdev_ioctl+0x1e0/0x610 [zfs]
  [  846.907622]  do_vfs_ioctl+0xa8/0x630
  [  846.907624]  ? handle_mm_fault+0xb1/0x1f0
  [  846.907625]  ? __do_page_fault+0x270/0x4d0
  [  846.907627]  SyS_ioctl+0x79/0x90
  [  846.907628]  do_syscall_64+0x73/0x130
  [  846.907630]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
  [  846.907631] RIP: 0033:0x7fc2734075d7
  [  846.907631] RSP: 002b:00007fff653a4b08 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
  [  846.907633] RAX: ffffffffffffffda RBX: 00007fff653a4b30 RCX: 00007fc2734075d7
  [  846.907633] RDX: 00007fff653a4b30 RSI: 0000000000005a12 RDI: 0000000000000003
  [  846.907634] RBP: 000055f3576e9660 R08: 000000000000ffff R09: 00007fc27345fed0
  [  846.907635] R10: 2f746c7561666564 R11: 0000000000000246 R12: 000055f3576e9660
  [  846.907637] R13: 00007fff653aaec0 R14: 00007fc2749036e0 R15: 00007fff653a8220

  Contents of "/proc/spl/kstat/zfs/arcstats"

  13 1 0x01 96 4608 11300672527 1808059980062
  name                            type data
  hits                            4    44186496
  misses                          4    1247761
  demand_data_hits                4    3327097
  demand_data_misses              4    17953
  demand_metadata_hits            4    39832270
  demand_metadata_misses          4    1076950
  prefetch_data_hits              4    1357
  prefetch_data_misses            4    23
  prefetch_metadata_hits          4    1025772
  prefetch_metadata_misses        4    152835
  mru_hits                        4    18193851
  mru_ghost_hits                  4    0
  mfu_hits                        4    24976976
  mfu_ghost_hits                  4    0
  deleted                         4    10
  mutex_miss                      4    0
  access_skip                     4    68
  evict_skip                      4    1
  evict_not_enough                4    0
  evict_l2_cached                 4    0
  evict_l2_eligible               4    101376
  evict_l2_ineligible             4    2048
  evict_l2_skip                   4    0
  hash_elements                   4    38683
  hash_elements_max               4    57741
  hash_collisions                 4    1520
  hash_chains                     4    5
  hash_chain_max                  4    1
  p                               4    16815604736
  c                               4    33631209472
  c_min                           4    2101950592
  c_max                           4    33631209472
  size                            4    833199872
  compressed_size                 4    286609408
  uncompressed_size               4    831992320
  overhead_size                   4    330568192
  hdr_size                        4    14660144
  data_size                       4    193579520
  metadata_size                   4    423598080
  dbuf_size                       4    46065424
  dnode_size                      4    112384064
  bonus_size                      4    42912640
  anon_size                       4    36502016
  anon_evictable_data             4    0
  anon_evictable_metadata         4    0
  mru_size                        4    310932480
  mru_evictable_data              4    27623424
  mru_evictable_metadata          4    136801792
  mru_ghost_size                  4    0
  mru_ghost_evictable_data        4    0
  mru_ghost_evictable_metadata    4    0
  mfu_size                        4    269743104
  mfu_evictable_data              4    53612032
  mfu_evictable_metadata          4    3195392
  mfu_ghost_size                  4    0
  mfu_ghost_evictable_data        4    0
  mfu_ghost_evictable_metadata    4    0
  l2_hits                         4    0
  l2_misses                       4    0
  l2_feeds                        4    0
  l2_rw_clash                     4    0
  l2_read_bytes                   4    0
  l2_write_bytes                  4    0
  l2_writes_sent                  4    0
  l2_writes_done                  4    0
  l2_writes_error                 4    0
  l2_writes_lock_retry            4    0
  l2_evict_lock_retry             4    0
  l2_evict_reading                4    0
  l2_evict_l1cached               4    0
  l2_free_on_write                4    0
  l2_abort_lowmem                 4    0
  l2_cksum_bad                    4    0
  l2_io_error                     4    0
  l2_size                         4    0
  l2_asize                        4    0
  l2_hdr_size                     4    0
  memory_throttle_count           4    0
  memory_direct_count             4    0
  memory_indirect_count           4    0
  memory_all_bytes                4    67262418944
  memory_free_bytes               4    57157578752
  memory_available_bytes          3    56106606592
  arc_no_grow                     4    0
  arc_tempreserve                 4    0
  arc_loaned_bytes                4    0
  arc_prune                       4    0
  arc_meta_used                   4    639620352
  arc_meta_limit                  4    25223407104
  arc_dnode_limit                 4    2522340710
  arc_meta_max                    4    965175896
  arc_meta_min                    4    16777216
  sync_wait_for_async             4    168
  demand_hit_predictive_prefetch  4    999
  arc_need_free                   4    0
  arc_sys_free                    4    1050975296

  Command "slabtop -o"

   Active / Total Objects (% used)    : 28354235 / 29140626 (97.3%)
   Active / Total Slabs (% used)      : 382017 / 382017 (100.0%)
   Active / Total Caches (% used)     : 95 / 128 (74.2%)
   Active / Total Size (% used)       : 4580408.56K / 4743868.00K (96.6%)
   Minimum / Average / Maximum Object : 0.01K / 0.16K / 21.81K

    OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
  17206144 17204917   0%    0.03K 134423      128    537692K kmalloc-32
  1141392 966790   0%    0.19K  27176       42    217408K dentry
  1114496 1004537   0%    0.06K  17414       64     69656K kmalloc-64
  1088192 1078186   0%    0.50K  17003       64    544096K kmalloc-512
  1038300 714247   0%    0.13K  17305       60    138440K kernfs_node_cache
  937536 931296   0%    0.25K  14649       64    234384K filp
  684160 682244   0%    0.06K  10690       64     42760K pid
  582099 569877   0%    0.59K  10983       53    351456K inode_cache
  520104 518529   0%    0.20K  13336       39    106688K vm_area_struct
  404334 388371   0%    0.09K   9627       42     38508K kmalloc-96
  342286 341596   0%    0.09K   7441       46     29764K anon_vma
  342016 338948   0%    0.25K   5344       64     85504K kmalloc-256
  277032 276822   0%    0.19K   6596       42     52768K cred_jar
  248352 241634   0%    0.66K   5174       48    165568K proc_inode_cache
  248320 233052   0%    0.01K    485      512      1940K kmalloc-8
  214984 143177   0%    0.57K   3839       56    122848K radix_tree_node

To manage notifications about this bug go to:
https://bugs.launchpad.net/linux/+bug/1781364/+subscriptions