group.of.nepali.translators team mailing list archive

[Tim Gardner <tim.gardner@xxxxxxxxxxxxx>]

Merged in v4.8

** Also affects: linux (Ubuntu Yakkety)
   Importance: High
     Assignee: Canonical Kernel Team (canonical-kernel-team)
       Status: Triaged

** Also affects: linux (Ubuntu Xenial)
   Importance: Undecided
       Status: New

** Changed in: linux (Ubuntu Yakkety)
       Status: Triaged => Fix Released

** Changed in: linux (Ubuntu Yakkety)
     Assignee: Canonical Kernel Team (canonical-kernel-team) => (unassigned)

** Changed in: linux (Ubuntu Xenial)
       Status: New => In Progress

** Changed in: linux (Ubuntu Xenial)
     Assignee: (unassigned) => Tim Gardner (timg-tpi)

-- 
You received this bug notification because you are a member of नेपाली
भाषा समायोजकहरुको समूह, which is subscribed to Xenial.
Matching subscriptions: Ubuntu 16.04 Bugs
https://bugs.launchpad.net/bugs/1620317

Title:
  ISST-LTE:pNV: system ben is hung during ST (nvme)

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Xenial:
  In Progress
Status in linux source package in Yakkety:
  Fix Released

Bug description:
  On when we are running I/O intensive tasks and CPU addition/removal,
  the block may hang stalling the entire machine.

  The backtrace below is one of the symptoms:

  [12747.111149] ---[ end trace b4d8d720952460b5 ]---
  [12747.126885] Trying to free IRQ 357 from IRQ context!
  [12747.146930] ------------[ cut here ]------------
  [12747.166674] WARNING: at /build/linux-iLHNl3/linux-4.4.0/kernel/irq/manage.c:1438
  [12747.184069] Modules linked in: minix nls_iso8859_1 rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) configfs ib_ipoib(OE) ib_cm(OE) ib_uverbs(OE) ib_umad(OE) mlx5_ib(OE) mlx4_ib(OE) ib_sa(OE) ib_mad(OE) ib_core(OE) ib_addr(OE) mlx4_en(OE) mlx4_core(OE) binfmt_misc xfs joydev input_leds mac_hid ofpart cmdlinepart powernv_flash ipmi_powernv mtd ipmi_msghandler at24 opal_prd powernv_rng ibmpowernv uio_pdrv_genirq uio sunrpc knem(OE) autofs4 btrfs xor raid6_pq hid_generic usbhid hid uas usb_storage nouveau ast bnx2x i2c_algo_bit ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops mlx5_core(OE) ahci drm mdio libcrc32c mlx_compat(OE) libahci vxlan nvme ip6_udp_tunnel udp_tunnel
  [12747.349013] CPU: 80 PID: 0 Comm: swapper/80 Tainted: G        W  OEL  4.4.0-21-generic #37-Ubuntu
  [12747.369046] task: c000000f1fab89b0 ti: c000000f1fb6c000 task.ti: c000000f1fb6c000
  [12747.404848] NIP: c000000000131888 LR: c000000000131884 CTR: 00000000300303f0
  [12747.808333] REGS: c000000f1fb6e550 TRAP: 0700   Tainted: G        W  OEL   (4.4.0-21-generic)
  [12747.867658] MSR: 9000000100029033 <SF,HV,EE,ME,IR,DR,RI,LE>  CR: 28022222  XER: 20000000
  [12747.884783] CFAR: c000000000aea8f4 SOFTE: 1 
  GPR00: c000000000131884 c000000f1fb6e7d0 c0000000015b4200 0000000000000028 
  GPR04: c000000f2a409c50 c000000f2a41b4e0 0000000f29480000 00000000000033da 
  GPR08: 0000000000000007 c000000000f8b27c 0000000f29480000 9000000100001003 
  GPR12: 0000000000002200 c000000007b6f800 c000000f2a40a938 0000000000000100 
  GPR16: c000000f11480000 0000000000003a98 0000000000000000 0000000000000000 
  GPR20: 0000000000000000 d000000009521008 d0000000095146a0 fffffffffffff000 
  GPR24: c000000004a19ef0 0000000000000000 0000000000000003 000000000000007d 
  GPR28: 0000000000000165 c000000eefeb1800 c000000eef830600 0000000000000165 
  [12748.243270] NIP [c000000000131888] __free_irq+0x238/0x370
  [12748.254089] LR [c000000000131884] __free_irq+0x234/0x370
  [12748.269738] Call Trace:
  [12748.286740] [c000000f1fb6e7d0] [c000000000131884] __free_irq+0x234/0x370 (unreliable)
  [12748.289687] [c000000f1fb6e860] [c000000000131af8] free_irq+0x88/0xb0
  [12748.304594] [c000000f1fb6e890] [d000000009514528] nvme_suspend_queue+0xc8/0x150 [nvme]
  [12748.333825] [c000000f1fb6e8c0] [d00000000951681c] nvme_dev_disable+0x3fc/0x400 [nvme]
  [12748.340913] [c000000f1fb6e9a0] [d000000009516ae4] nvme_timeout+0xe4/0x260 [nvme]
  [12748.357136] [c000000f1fb6ea60] [c000000000548a34] blk_mq_rq_timed_out+0x64/0x110
  [12748.383939] [c000000f1fb6ead0] [c00000000054c540] bt_for_each+0x160/0x170
  [12748.399292] [c000000f1fb6eb40] [c00000000054d4e8] blk_mq_queue_tag_busy_iter+0x78/0x110
  [12748.402665] [c000000f1fb6eb90] [c000000000547358] blk_mq_rq_timer+0x48/0x140
  [12748.438649] [c000000f1fb6ebd0] [c00000000014a13c] call_timer_fn+0x5c/0x1c0
  [12748.468126] [c000000f1fb6ec60] [c00000000014a5fc] run_timer_softirq+0x31c/0x3f0
  [12748.483367] [c000000f1fb6ed30] [c0000000000beb78] __do_softirq+0x188/0x3e0
  [12748.498378] [c000000f1fb6ee20] [c0000000000bf048] irq_exit+0xc8/0x100
  [12748.501048] [c000000f1fb6ee40] [c00000000001f954] timer_interrupt+0xa4/0xe0
  [12748.516377] [c000000f1fb6ee70] [c000000000002714] decrementer_common+0x114/0x180
  [12748.547282] --- interrupt: 901 at arch_local_irq_restore+0x74/0x90
  [12748.547282]     LR = arch_local_irq_restore+0x74/0x90
  [12748.574141] [c000000f1fb6f160] [0000000000000001] 0x1 (unreliable)
  [12748.592405] [c000000f1fb6f180] [c000000000aedc3c] dump_stack+0xd0/0xf0
  [12748.596461] [c000000f1fb6f1c0] [c0000000001006fc] dequeue_task_idle+0x5c/0x90
  [12748.611532] [c000000f1fb6f230] [c0000000000f6080] deactivate_task+0xc0/0x130
  [12748.627685] [c000000f1fb6f270] [c000000000adcb10] __schedule+0x440/0x990
  [12748.654416] [c000000f1fb6f300] [c000000000add0a8] schedule+0x48/0xc0
  [12748.670558] [c000000f1fb6f330] [c000000000ae1474] schedule_timeout+0x274/0x350
  [12748.673485] [c000000f1fb6f420] [c000000000ade23c] wait_for_common+0xec/0x240
  [12748.699192] [c000000f1fb6f4a0] [c0000000000e6908] kthread_stop+0x88/0x210
  [12748.718385] [c000000f1fb6f4e0] [d000000009514240] nvme_dev_list_remove+0x90/0x110 [nvme]
  [12748.748925] [c000000f1fb6f510] [d000000009516498] nvme_dev_disable+0x78/0x400 [nvme]
  [12748.752112] [c000000f1fb6f5f0] [d000000009516ae4] nvme_timeout+0xe4/0x260 [nvme]
  [12748.775395] [c000000f1fb6f6b0] [c000000000548a34] blk_mq_rq_timed_out+0x64/0x110
  [12748.821069] [c000000f1fb6f720] [c00000000054c540] bt_for_each+0x160/0x170
  [12748.851733] [c000000f1fb6f790] [c00000000054d4e8] blk_mq_queue_tag_busy_iter+0x78/0x110
  [12748.883093] [c000000f1fb6f7e0] [c000000000547358] blk_mq_rq_timer+0x48/0x140
  [12748.918348] [c000000f1fb6f820] [c00000000014a13c] call_timer_fn+0x5c/0x1c0
  [12748.934743] [c000000f1fb6f8b0] [c00000000014a5fc] run_timer_softirq+0x31c/0x3f0
  [12748.938084] [c000000f1fb6f980] [c0000000000beb78] __do_softirq+0x188/0x3e0
  [12748.960815] [c000000f1fb6fa70] [c0000000000bf048] irq_exit+0xc8/0x100
  [12748.992175] [c000000f1fb6fa90] [c00000000001f954] timer_interrupt+0xa4/0xe0
  [12749.019299] [c000000f1fb6fac0] [c000000000002714] decrementer_common+0x114/0x180
  [12749.037168] --- interrupt: 901 at arch_local_irq_restore+0x74/0x90
  [12749.037168]     LR = arch_local_irq_restore+0x74/0x90
  [12749.079044] [c000000f1fb6fdb0] [c000000f2a41d680] 0xc000000f2a41d680 (unreliable)
  [12749.081736] [c000000f1fb6fdd0] [c000000000909a28] cpuidle_enter_state+0x1a8/0x410
  [12749.127094] [c000000f1fb6fe30] [c000000000119a88] call_cpuidle+0x78/0xd0
  [12749.144435] [c000000f1fb6fe70] [c000000000119e5c] cpu_startup_entry+0x37c/0x480
  [12749.166156] [c000000f1fb6ff30] [c00000000004563c] start_secondary+0x33c/0x360
  [12749.186929] [c000000f1fb6ff90] [c000000000008b6c] start_secondary_prolog+0x10/0x14
  [12749.223828] Instruction dump:
  [12749.223856] 4e800020 4bf83a5d 60000000 4bffff64 4bf83a51 60000000 4bffffa8 3c62ff7b 
  [12749.233245] 7f84e378 38630fe0 489b900d 60000000 <0fe00000> 4bfffe20 7d2903a6 387d0118 
  [12749.298371] ---[ end trace b4d8d720952460b6 ]---

  == Comment: #184 - Gabriel Krisman Bertazi <gbertazi@xxxxxxxxxx> - 2016-07-29 12:55:48 ==
  I got it figured out.  The nvme driver is not playing nice with the block timeout infrastructure, in a way that the timeout code goes into a live lock, waiting for the queue to be released.  CPU hotplug, on the other hand, who is holding the queue freeze lock at the time, is waiting for an outstanding request to timeout (or complete).  This request, in turn is stuck in the device, requiring a reset triggered by a timeout, which never happens due to the live lock.

  I don't have the reason why the request is stuck inside the device
  requiring a timeout, but this could even be caused by the Leaf
  firmware itself. I also see some successful timeouts  triggered under
  normal conditions.  In the failure event, we should be able to abort
  the request normally, but this happens via the timeout infrastructure,
  which is blocked during cpu hotplug events.

  I have a quirk to fully recover after the failure, by forcing a reset
  of the stucked IO, which allows the cpu hotplug completion and block
  layer recovery.  I have a machine hitting the failure every few
  minutes in a loop, and recovering from it with my patch.

  Patch submitted to linux-block

  https://marc.info/?l=linux-block&m=146976739016592&w=2

  == Comment: #207 - Gabriel Krisman Bertazi <gbertazi@xxxxxxxxxx> - 2016-09-05 09:13:51 ==
  Canonical,

  This is fixed by:

  e57690fe009b ("blk-mq: don't overwrite rq->mq_ctx")
  0e87e58bf60e ("blk-mq: improve warning for running a queue on the wrong CPU")
  71f79fb3179e (" blk-mq: Allow timeouts to run while queue is freezing")

  Which will apply cleanly on top of your kernel.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1620317/+subscriptions