lttng team mailing list archive

Thread
Date
[Bug 1846485] Re: lttng causes crash with 5.0 based kernel

To: lttng@xxxxxxxxxxxxxxxxxxx
From: Matthew Ruffell <1846485@xxxxxxxxxxxxxxxxxx>
Date: Thu, 03 Oct 2019 23:43:19 -0000
Reply-to: Bug 1846485 <1846485@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Also affects: lttng-modules (Ubuntu Bionic)
   Importance: Undecided
       Status: New

** Changed in: lttng-modules (Ubuntu Bionic)
       Status: New => Fix Committed

** Changed in: lttng-modules (Ubuntu)
       Status: New => Fix Released

** Changed in: lttng-modules (Ubuntu Bionic)
     Assignee: (unassigned) => Matthew Ruffell (mruffell)

** Tags added: sts

-- 
You received this bug notification because you are a member of Ubuntu
LTTng, which is subscribed to lttng-modules in Ubuntu.
Matching subscriptions: lttng-modules-bugs
https://bugs.launchpad.net/bugs/1846485

Title:
  lttng causes crash with 5.0 based kernel

Status in lttng-modules package in Ubuntu:
  Fix Released
Status in lttng-modules source package in Bionic:
  Fix Committed

Bug description:
  There is a regression with the 5.0 based kernel.  The following is the
  good and bad kernels:

  OS                           Status      Kernel veriosn
  18.04.201906170 BAD  Linux nt1000000 5.0.0-1014-azure
  18.04.201906040 GOOD Linux nt1000000 4.18.0-1020-azure

  The crash can be reproduced with this line:
  sudo apt-get install lttng-tools lttng-modules-dkms liblttng-ust0


  
  The following trace occurs

  2019/09/26 17:36:30.129283 INFO ExtHandler Stopped tracking cgroup: Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029, path: /sys/fs/cgroup/cpu/system.slice/Microsoft.OSTCExtensions.LinuxDiagnostic_2.3.9029_fa19de8a-0c72-449e-9754-451dcef4340e.scope
  2019/09/26 17:36:30.130284 INFO ExtHandler Stopped tracking cgroup: Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029, path: /sys/fs/cgroup/memory/system.slice/Microsoft.OSTCExtensions.LinuxDiagnostic_2.3.9029_fa19de8a-0c72-449e-9754-451dcef4340e.scope
  [  346.729298] PKCS#7 signature not signed with a trusted key
  [  346.735782] PKCS#7 signature not signed with a trusted key
  [  346.744261] PKCS#7 signature not signed with a trusted key
  [  346.750132] PKCS#7 signature not signed with a trusted key
  [  346.755284] PKCS#7 signature not signed with a trusted key
  [  346.768389] PKCS#7 signature not signed with a trusted key
  [  346.778232] BUG: unable to handle kernel paging request at 00000000000032a0
  [  346.778623] #PF error: [normal kernel read fault]
  [  346.778623] PGD 0 P4D 0
  [  346.778623] Oops: 0000 [#1] SMP PTI
  [  346.778623] CPU: 1 PID: 26741 Comm: lttng-sessiond Kdump: loaded Tainted: G           OE     5.0.0-1018-azure #19~18.04.1-Ubuntu
  [  346.778623] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090007  06/02/2017
  [  346.778623] RIP: 0010:lttng_tracepoint_notify+0x172/0x210 [lttng_tracer]
  [  346.778623] Code: eb 1a 49 39 c6 0f 85 ab 00 00 00 49 8b 55 10 41 83 c4 01 44 39 a2 8c 02 00 00 76 a3 48 8b 92 90 02 00 00 49 63 c4 4c 63 34 82 <49> 8b 1e 48 89 df e8 33 fbff ff 48 85 c0 49 89 c7 74 52 49 8b 47
  [  346.795750] RSP: 0018:ffff9d5382ecbbd8 EFLAGS: 00010246
  [  346.795750] RAX: 0000000000000000 RBX: ffff9183eb560040 RCX: 0000000000000005
  [  346.795750] RDX: ffffffffc0042420 RSI: 0000000000000001 RDI: ffffffffc0739100
  [  346.795750] RBP: ffff9d5382ecbc08 R08: ffff9183f6c660c0 R09: ffff9183f6807600
  [  346.795750] R10: ffff9d5382ecbb48 R11: 00000000ffffffff R12: 0000000000000000
  [  346.795750] R13: ffff9183eb560040 R14: 00000000000032a0 R15: ffffffffc0739740
  [  346.795750] FS:  00007fd33176c980(0000) GS:ffff9183f6c40000(0000) knlGS:0000000000000000
  [  346.795750] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  [  346.795750] CR2: 00000000000032a0 CR3: 000000025dec6003 CR4: 00000000001606e0
  [  346.795750] Call Trace:
  [  346.795750]  register_tracepoint_module_notifier+0x57/0x80
  [  346.795750]  ? 0xffffffffc0778000
  [  346.795750]  lttng_tracepoint_init+0x45/0xc40 [lttng_tracer]
  [  346.795750]  lttng_events_init+0xbc/0x233 [lttng_tracer]
  [  346.795750]  ? 0xffffffffc0778000
  [  346.795750]  do_one_initcall+0x4a/0x1be
  [  346.795750]  ? _cond_resched+0x19/0x40
  [  346.795750]  ? kmem_cache_alloc_trace+0x151/0x1c0
  [  346.795750]  do_init_module+0x5f/0x216
  [  346.795750]  load_module+0x19f6/0x20a0
  [  346.795750]  __do_sys_finit_module+0xfc/0x120
  [  346.795750]  ? __do_sys_finit_module+0xfc/0x120
  [  346.795750]  __x64_sys_finit_module+0x1a/0x20
  [  346.795750]  do_syscall_64+0x61/0x190
  [  346.795750]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
  [  346.795750] RIP: 0033:0x7fd32f9d7839
  [  346.795750] Code: 00 f3 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c348 8b 0d 1f f6 2c 00 f7 d8 64 89 01 48
  [  346.795750] RSP: 002b:00007fff8a413f88 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
  [  346.795750] RAX: ffffffffffffffda RBX: 000055e7fc7d1800 RCX: 00007fd32f9d7839
  [  346.795750] RDX: 0000000000000000 RSI: 00007fd331354145 RDI: 0000000000000018
  [  346.795750] RBP: 00007fd331354145 R08: 0000000000000000 R09: 000055e7fc7d5ac0
  [  346.795750] R10: 0000000000000018 R11: 0000000000000246 R12: 0000000000000000
  [  346.795750] R13: 000055e7fc7d6120 R14: 0000000000000000 R15: 000055e7fc7d5ac0
  [  346.795750] Modules linked in: lttng_tracer(OE+) lttng_statedump(OE) lttng_kprobes(OE) lttng_clock(OE) lttng_lib_ring_buffer(OE) lttng_kretprobes(OE) xt_owner xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_security bpfilter nls_iso8859_1 serio_raw hv_balloon joydev sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel hid_generic aesni_intel aes_x86_64 crypto_simd cryptd glue_helper hid_hyperv hyperv_keyboardhid hv_utils hv_netvsc hyperv_fb cfbfillrect cfbimgblt cfbcopyarea pata_acpi
  [  346.795750] CR2: 00000000000032a0
  [    0.000000] Linux version 5.0.0-1018-azure (buildd@lgw01-amd64-048) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) #19~18.04.1-Ubuntu SMP Wed Aug 21 05:13:05 UTC 2019 (Ubuntu 5.0.0-1018.19~18.04.1-azure 5.0.21)
  [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.0.0-1018-azure root=UUID=7b4b88a7-6d85-44eb-9155-3132fd3d1cd8 ro console=tty1 console=ttyS0 earlyprintk=ttyS0 rootdelay=300 nr_cpus=1 systemd.unit=kdump-tools-dump.service irqpoll nousb ata_piix.prefer_ms_hyperv=0 elfcorehdr=867700K
  [    0.000000] KERNEL supported cpus:
  [    0.000000]   Intel GenuineIntel
  [    0.000000]   AMD AuthenticAMD
  [    0.000000]   Hygon HygonGenuine
  [    0.000000]   Centaur CentaurHauls
  [    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
  [    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
  [    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
  [    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
  [    0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
  [    0.000000] BIOS-provided physical RAM map:
  [    0.000000] BIOS-e820: [mem 0x0000000000001000-0x000000000009fbff] usable
  [    0.000000] BIOS-e820: [mem 0x0000000029000000-0x0000000034f5cfff] usable
  [    0.000000] BIOS-e820: [mem 0x0000000034fffc00-0x0000000034ffffff] usable
  [    0.000000] BIOS-e820: [mem 0x000000003fff0000-0x000000003fffefff] ACPI data
  [    0.000000] BIOS-e820: [mem 0x000000003ffff000-0x000000003fffffff] ACPI NVS
  [    0.000000] printk: bootconsole [earlyser0] enabled
  [    0.000000] NX (Execute Disable) protection: active
  [    0.000000] SMBIOS 2.3 present.
  [    0.000000] DMI: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090007  06/02/2017
  [    0.000000] Hypervisor detected: Microsoft Hyper-V
  [    0.000000] Hyper-V: features 0x2e7f, hints 0x40c2c
  [    0.000000] Hyper-V Host Build:14393-10.0-0-0.299
  [    0.000000] Hyper-V: LAPIC Timer Frequency: 0xc3500
  [    0.000000] tsc: Marking TSC unstable due to running on Hyper-V
  [    0.000000] Hyper-V: Using hypercall for remote TLB flush
  [    0.000000] tsc: Detected 2394.455 MHz processor
  [    0.001399] last_pfn = 0x35000 max_arch_pfn = 0x400000000
  [    0.004043] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT
  Memory KASLR using RDRAND RDTSC...
  [    0.024144] found SMP MP-table at [mem 0x000ff780-0x000ff78f]
  [    0.026977] check: Scanning 1 areas for low memory corruption
  [    0.029761] Using GB pages for direct mapping
  [    0.032036] RAMDISK: [mem 0x32625000-0x331fffff]
  [    0.034267] ACPI: Early table checksum verification disabled
  [    0.037090] ACPI: RSDP 0x00000000000F5BF0 000014 (v00 ACPIAM)
  [    0.039763] ACPI: RSDT 0x000000003FFF0000 000040 (v01 VRTUAL MICROSFT 06001702 MSFT 00000097)
  [    0.043785] ACPI: FACP 0x000000003FFF0200 000081 (v02 VRTUAL MICROSFT 06001702 MSFT 00000097)
  [    0.048319] ACPI: DSDT 0x000000003FFF1D24 003CBE (v01 MSFTVM MSFTVM02 00000002 INTL 02002026)
  [    0.052474] ACPI: FACS 0x000000003FFFF000 000040
  [    0.054736] ACPI: WAET 0x000000003FFF1A80 000028 (v01 VRTUAL MICROSFT 06001702 MSFT 00000097)
  [    0.058789] ACPI: SLIC 0x000000003FFF1AC0 000176 (v01 VRTUAL MICROSFT 06001702 MSFT 00000097)
  [    0.062902] ACPI: OEM0 0x000000003FFF1CC0 000064 (v01 VRTUAL MICROSFT 06001702 MSFT 00000097)
  [    0.066950] ACPI: SRAT 0x000000003FFF0800 000140 (v02 VRTUAL MICROSFT 00000001 MSFT 00000001)
  [    0.071109] ACPI: APIC 0x000000003FFF0300 000452 (v01 VRTUAL MICROSFT 06001702 MSFT 00000097)
  [    0.075184] ACPI: OEMB 0x000000003FFFF040 000064 (v01 VRTUAL MICROSFT 06001702 MSFT 00000097)
  [    0.079460] SRAT: PXM 0 -> APIC 0x00 -> Node 0
  [    0.081643] SRAT: PXM 0 -> APIC 0x01 -> Node 0
  [    0.083847] ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0x3fffffff] hotplug
  [    0.087160] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x27fffffff] hotplug
  [    0.090457] ACPI: SRAT: Node 0 PXM 0 [mem 0x280200000-0xfdfffffff] hotplug
  [    0.093834] ACPI: SRAT: Node 0 PXM 0 [mem 0x1000000000-0xffffffffff] hotplug
  [    0.097266] ACPI: SRAT: Node 0 PXM 0 [mem 0x10000200000-0x1ffffffffff] hotplug
  [    0.100827] ACPI: SRAT: Node 0 PXM 0 [mem 0x20000200000-0x3ffffffffff] hotplug
  [    0.109180] NODE_DATA(0) allocated [mem 0x34f32000-0x34f5cfff]
  [    0.112114] Zone ranges:
  [    0.113485]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
  [    0.116461]   DMA32    [mem 0x0000000001000000-0x0000000034ffffff]
  [    0.119510]   Normal   empty
  [    0.120892]   Device   empty
  [    0.122274] Movable zone start for each node
  [    0.124384] Early memory node ranges
  [    0.126361]   node   0: [mem 0x0000000000001000-0x000000000009efff]
  [    0.129406]   node   0: [mem 0x0000000029000000-0x0000000034f5cfff]
  [    0.133029] Zeroed struct page in unavailable ranges: 36869 pages
  [    0.133031] Initmem setup node 0 [mem 0x0000000000001000-0x0000000034f5cfff]
  [    0.143898] ACPI: PM-Timer IO Port: 0x408
  [    0.145927] APIC: NR_CPUS/possible_cpus limit of 1 almost reached. Keeping one slot for boot cpu.  Processor 0/0x0 ignored.
  [    0.151414] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
  [    0.154623] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23
  [    0.158081] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
  [    0.161257] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
  [    0.164541] Using ACPI (MADT) for SMP configuration information
  [    0.167650] smpboot: 128 Processors exceeds NR_CPUS limit of 1
  [    0.170659] smpboot: Allowing 1 CPUs, 0 hotplug CPUs
  [    0.173280] PM: Registered nosave memory: [mem 0x00000000-0x00000fff]
  [    0.176584] PM: Registered nosave memory: [mem 0x0009f000-0x28ffffff]
  [    0.179894] PM: Registered nosave memory: [mem 0x34f5d000-0x34ffffff]
  [    0.184693] [mem 0x40000000-0xffffffff] available for PCI devices
  [    0.212824] Booting paravirtualized kernel on bare hardware
  [    0.215671] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
  [    0.221827] random: get_random_bytes called from start_kernel+0x93/0x503 with crng_init=0
  [    0.226026] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1
  [    0.230128] percpu: Embedded 45 pages/cpu s147456 r8192 d28672 u2097152
  [    0.233727] Hyper-V: PV spinlocks enabled
  [    0.235483] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes)
  [    0.238597] Built 1 zonelists, mobility grouping on.  Total pages: 48357
  [    0.241718] Policy zone: DMA32
  [    0.243155] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.0.0-1018-azure root=UUID=7b4b88a7-6d85-44eb-9155-3132fd3d1cd8 ro console=tty1 console=ttyS0 earlyprintk=ttyS0 rootdelay=300 nr_cpus=1 systemd.unit=kdump-tools-dump.service irqpoll nousb ata_piix.prefer_ms_hyperv=0 elfcorehdr=867700K
  [    0.257201] Misrouted IRQ fixup and polling support enabled
  [    0.259980] This may significantly impact system performance
  [    0.272336] Memory: 82316K/196588K available (12292K kernel code, 2232K rwdata, 3840K rodata, 2352K init, 5536K bss, 114272K reserved, 0K cma-reserved)
  [    0.279360] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
  [    0.282719] Kernel/User page tables isolation: enabled
  [    0.285246] ftrace: allocating 38598 entries in 151 pages
  [    0.303379] rcu: Hierarchical RCU implementation.
  [    0.305670] rcu:     RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=1.
  [    0.308863]  Tasks RCU enabled.
  [    0.310334] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
  [    0.314124] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
  [    0.320349] NR_IRQS: 524544, nr_irqs: 256, preallocated irqs: 16
  [    0.323901] Spurious LAPIC timer interrupt on cpu 0
  [    0.330364] Console: colour VGA+ 80x25
  [    0.000000] Linux version 5.0.0-1018-azure (buildd@lgw01-amd64-048) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) #19~18.04.1-Ubuntu SMP Wed Aug 21 05:13:05 UTC 2019 (Ubuntu 5.0.0-1018.19~18.04.1-azure 5.0.21)
  [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.0.0-1018-azure root=UUID=7b4b88a7-6d85-44eb-9155-3132fd3d1cd8 ro console=tty1 console=ttyS0 earlyprintk=ttyS0 rootdelay=300 nr_cpus=1 systemd.unit=kdump-tools-dump.service irqpoll nousb ata_piix.prefer_ms_hyperv=0 elfcorehdr=867700K
  [    0.000000] KERNEL supported cpus:
  [    0.000000]   Intel GenuineIntel
  [    0.000000]   AMD AuthenticAMD
  [    0.000000]   Hygon HygonGenuine
  [    0.000000]   Centaur CentaurHauls
  [    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
  [    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
  [    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
  [    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
  [    0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
  [    0.000000] BIOS-provided physical RAM map:
  [    0.000000] BIOS-e820: [mem 0x0000000000001000-0x000000000009fbff] usable
  [    0.000000] BIOS-e820: [mem 0x0000000029000000-0x0000000034f5cfff] usable
  [    0.000000] BIOS-e820: [mem 0x0000000034fffc00-0x0000000034ffffff] usable
  [    0.000000] BIOS-e820: [mem 0x000000003fff0000-0x000000003fffefff] ACPI data
  [    0.000000] BIOS-e820: [mem 0x000000003ffff000-0x000000003fffffff] ACPI NVS
  [    0.000000] printk: bootconsole [earlyser0] enabled
  [    0.000000] NX (Execute Disable) protection: active
  [    0.000000] SMBIOS 2.3 present.
  [    0.000000] DMI: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090007  06/02/2017
  [    0.000000] Hypervisor detected: Microsoft Hyper-V
  [    0.000000] Hyper-V: features 0x2e7f, hints 0x40c2c
  [    0.000000] Hyper-V Host Build:14393-10.0-0-0.299
  [    0.000000] Hyper-V: LAPIC Timer Frequency: 0xc3500
  [    0.000000] tsc: Marking TSC unstable due to running on Hyper-V
  [    0.000000] Hyper-V: Using hypercall for remote TLB flush
  [    0.000000] tsc: Detected 2394.455 MHz processor
  [    0.001399] last_pfn = 0x35000 max_arch_pfn = 0x400000000
  [    0.004043] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT
  [    0.024144] found SMP MP-table at [mem 0x000ff780-0x000ff78f]
  [    0.026977] check: Scanning 1 areas for low memory corruption
  [    0.029761] Using GB pages for direct mapping
  [    0.032036] RAMDISK: [mem 0x32625000-0x331fffff]
  [    0.034267] ACPI: Early table checksum verification disabled
  [    0.037090] ACPI: RSDP 0x00000000000F5BF0 000014 (v00 ACPIAM)
  [    0.039763] ACPI: RSDT 0x000000003FFF0000 000040 (v01 VRTUAL MICROSFT 06001702 MSFT 00000097)
  [    0.043785] ACPI: FACP 0x000000003FFF0200 000081 (v02 VRTUAL MICROSFT 06001702 MSFT 00000097)
  [    0.048319] ACPI: DSDT 0x000000003FFF1D24 003CBE (v01 MSFTVM MSFTVM02 00000002 INTL 02002026)
  [    0.052474] ACPI: FACS 0x000000003FFFF000 000040
  [    0.054736] ACPI: WAET 0x000000003FFF1A80 000028 (v01 VRTUAL MICROSFT 06001702 MSFT 00000097)
  [    0.058789] ACPI: SLIC 0x000000003FFF1AC0 000176 (v01 VRTUAL MICROSFT 06001702 MSFT 00000097)
  [    0.062902] ACPI: OEM0 0x000000003FFF1CC0 000064 (v01 VRTUAL MICROSFT 06001702 MSFT 00000097)
  [    0.066950] ACPI: SRAT 0x000000003FFF0800 000140 (v02 VRTUAL MICROSFT 00000001 MSFT 00000001)
  [    0.071109] ACPI: APIC 0x000000003FFF0300 000452 (v01 VRTUAL MICROSFT 06001702 MSFT 00000097)
  [    0.075184] ACPI: OEMB 0x000000003FFFF040 000064 (v01 VRTUAL MICROSFT 06001702 MSFT 00000097)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/lttng-modules/+bug/1846485/+subscriptions