lttng team mailing list archive

Thread
Date
[Bug 1846485] [NEW] lttng causes crash with 5.0 based kernel

To: lttng@xxxxxxxxxxxxxxxxxxx
From: Launchpad Bug Tracker <1846485@xxxxxxxxxxxxxxxxxx>
Date: Thu, 03 Oct 2019 21:14:22 -0000
Reply-to: Bug 1846485 <1846485@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
You have been subscribed to a public bug:

There is a regression with the 5.0 based kernel.  The following is the
good and bad kernels:

OS                           Status      Kernel veriosn
18.04.201906170 BAD  Linux nt1000000 5.0.0-1014-azure
18.04.201906040 GOOD Linux nt1000000 4.18.0-1020-azure

The crash can be reproduced with this line:
sudo apt-get install lttng-tools lttng-modules-dkms liblttng-ust0



The following trace occurs

2019/09/26 17:36:30.129283 INFO ExtHandler Stopped tracking cgroup: Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029, path: /sys/fs/cgroup/cpu/system.slice/Microsoft.OSTCExtensions.LinuxDiagnostic_2.3.9029_fa19de8a-0c72-449e-9754-451dcef4340e.scope
2019/09/26 17:36:30.130284 INFO ExtHandler Stopped tracking cgroup: Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029, path: /sys/fs/cgroup/memory/system.slice/Microsoft.OSTCExtensions.LinuxDiagnostic_2.3.9029_fa19de8a-0c72-449e-9754-451dcef4340e.scope
[  346.729298] PKCS#7 signature not signed with a trusted key
[  346.735782] PKCS#7 signature not signed with a trusted key
[  346.744261] PKCS#7 signature not signed with a trusted key
[  346.750132] PKCS#7 signature not signed with a trusted key
[  346.755284] PKCS#7 signature not signed with a trusted key
[  346.768389] PKCS#7 signature not signed with a trusted key
[  346.778232] BUG: unable to handle kernel paging request at 00000000000032a0
[  346.778623] #PF error: [normal kernel read fault]
[  346.778623] PGD 0 P4D 0
[  346.778623] Oops: 0000 [#1] SMP PTI
[  346.778623] CPU: 1 PID: 26741 Comm: lttng-sessiond Kdump: loaded Tainted: G           OE     5.0.0-1018-azure #19~18.04.1-Ubuntu
[  346.778623] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090007  06/02/2017
[  346.778623] RIP: 0010:lttng_tracepoint_notify+0x172/0x210 [lttng_tracer]
[  346.778623] Code: eb 1a 49 39 c6 0f 85 ab 00 00 00 49 8b 55 10 41 83 c4 01 44 39 a2 8c 02 00 00 76 a3 48 8b 92 90 02 00 00 49 63 c4 4c 63 34 82 <49> 8b 1e 48 89 df e8 33 fbff ff 48 85 c0 49 89 c7 74 52 49 8b 47
[  346.795750] RSP: 0018:ffff9d5382ecbbd8 EFLAGS: 00010246
[  346.795750] RAX: 0000000000000000 RBX: ffff9183eb560040 RCX: 0000000000000005
[  346.795750] RDX: ffffffffc0042420 RSI: 0000000000000001 RDI: ffffffffc0739100
[  346.795750] RBP: ffff9d5382ecbc08 R08: ffff9183f6c660c0 R09: ffff9183f6807600
[  346.795750] R10: ffff9d5382ecbb48 R11: 00000000ffffffff R12: 0000000000000000
[  346.795750] R13: ffff9183eb560040 R14: 00000000000032a0 R15: ffffffffc0739740
[  346.795750] FS:  00007fd33176c980(0000) GS:ffff9183f6c40000(0000) knlGS:0000000000000000
[  346.795750] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  346.795750] CR2: 00000000000032a0 CR3: 000000025dec6003 CR4: 00000000001606e0
[  346.795750] Call Trace:
[  346.795750]  register_tracepoint_module_notifier+0x57/0x80
[  346.795750]  ? 0xffffffffc0778000
[  346.795750]  lttng_tracepoint_init+0x45/0xc40 [lttng_tracer]
[  346.795750]  lttng_events_init+0xbc/0x233 [lttng_tracer]
[  346.795750]  ? 0xffffffffc0778000
[  346.795750]  do_one_initcall+0x4a/0x1be
[  346.795750]  ? _cond_resched+0x19/0x40
[  346.795750]  ? kmem_cache_alloc_trace+0x151/0x1c0
[  346.795750]  do_init_module+0x5f/0x216
[  346.795750]  load_module+0x19f6/0x20a0
[  346.795750]  __do_sys_finit_module+0xfc/0x120
[  346.795750]  ? __do_sys_finit_module+0xfc/0x120
[  346.795750]  __x64_sys_finit_module+0x1a/0x20
[  346.795750]  do_syscall_64+0x61/0x190
[  346.795750]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[  346.795750] RIP: 0033:0x7fd32f9d7839
[  346.795750] Code: 00 f3 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c348 8b 0d 1f f6 2c 00 f7 d8 64 89 01 48
[  346.795750] RSP: 002b:00007fff8a413f88 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[  346.795750] RAX: ffffffffffffffda RBX: 000055e7fc7d1800 RCX: 00007fd32f9d7839
[  346.795750] RDX: 0000000000000000 RSI: 00007fd331354145 RDI: 0000000000000018
[  346.795750] RBP: 00007fd331354145 R08: 0000000000000000 R09: 000055e7fc7d5ac0
[  346.795750] R10: 0000000000000018 R11: 0000000000000246 R12: 0000000000000000
[  346.795750] R13: 000055e7fc7d6120 R14: 0000000000000000 R15: 000055e7fc7d5ac0
[  346.795750] Modules linked in: lttng_tracer(OE+) lttng_statedump(OE) lttng_kprobes(OE) lttng_clock(OE) lttng_lib_ring_buffer(OE) lttng_kretprobes(OE) xt_owner xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_security bpfilter nls_iso8859_1 serio_raw hv_balloon joydev sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel hid_generic aesni_intel aes_x86_64 crypto_simd cryptd glue_helper hid_hyperv hyperv_keyboardhid hv_utils hv_netvsc hyperv_fb cfbfillrect cfbimgblt cfbcopyarea pata_acpi
[  346.795750] CR2: 00000000000032a0
[    0.000000] Linux version 5.0.0-1018-azure (buildd@lgw01-amd64-048) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) #19~18.04.1-Ubuntu SMP Wed Aug 21 05:13:05 UTC 2019 (Ubuntu 5.0.0-1018.19~18.04.1-azure 5.0.21)
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.0.0-1018-azure root=UUID=7b4b88a7-6d85-44eb-9155-3132fd3d1cd8 ro console=tty1 console=ttyS0 earlyprintk=ttyS0 rootdelay=300 nr_cpus=1 systemd.unit=kdump-tools-dump.service irqpoll nousb ata_piix.prefer_ms_hyperv=0 elfcorehdr=867700K
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000]   AMD AuthenticAMD
[    0.000000]   Hygon HygonGenuine
[    0.000000]   Centaur CentaurHauls
[    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000001000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x0000000029000000-0x0000000034f5cfff] usable
[    0.000000] BIOS-e820: [mem 0x0000000034fffc00-0x0000000034ffffff] usable
[    0.000000] BIOS-e820: [mem 0x000000003fff0000-0x000000003fffefff] ACPI data
[    0.000000] BIOS-e820: [mem 0x000000003ffff000-0x000000003fffffff] ACPI NVS
[    0.000000] printk: bootconsole [earlyser0] enabled
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 2.3 present.
[    0.000000] DMI: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090007  06/02/2017
[    0.000000] Hypervisor detected: Microsoft Hyper-V
[    0.000000] Hyper-V: features 0x2e7f, hints 0x40c2c
[    0.000000] Hyper-V Host Build:14393-10.0-0-0.299
[    0.000000] Hyper-V: LAPIC Timer Frequency: 0xc3500
[    0.000000] tsc: Marking TSC unstable due to running on Hyper-V
[    0.000000] Hyper-V: Using hypercall for remote TLB flush
[    0.000000] tsc: Detected 2394.455 MHz processor
[    0.001399] last_pfn = 0x35000 max_arch_pfn = 0x400000000
[    0.004043] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT
Memory KASLR using RDRAND RDTSC...
[    0.024144] found SMP MP-table at [mem 0x000ff780-0x000ff78f]
[    0.026977] check: Scanning 1 areas for low memory corruption
[    0.029761] Using GB pages for direct mapping
[    0.032036] RAMDISK: [mem 0x32625000-0x331fffff]
[    0.034267] ACPI: Early table checksum verification disabled
[    0.037090] ACPI: RSDP 0x00000000000F5BF0 000014 (v00 ACPIAM)
[    0.039763] ACPI: RSDT 0x000000003FFF0000 000040 (v01 VRTUAL MICROSFT 06001702 MSFT 00000097)
[    0.043785] ACPI: FACP 0x000000003FFF0200 000081 (v02 VRTUAL MICROSFT 06001702 MSFT 00000097)
[    0.048319] ACPI: DSDT 0x000000003FFF1D24 003CBE (v01 MSFTVM MSFTVM02 00000002 INTL 02002026)
[    0.052474] ACPI: FACS 0x000000003FFFF000 000040
[    0.054736] ACPI: WAET 0x000000003FFF1A80 000028 (v01 VRTUAL MICROSFT 06001702 MSFT 00000097)
[    0.058789] ACPI: SLIC 0x000000003FFF1AC0 000176 (v01 VRTUAL MICROSFT 06001702 MSFT 00000097)
[    0.062902] ACPI: OEM0 0x000000003FFF1CC0 000064 (v01 VRTUAL MICROSFT 06001702 MSFT 00000097)
[    0.066950] ACPI: SRAT 0x000000003FFF0800 000140 (v02 VRTUAL MICROSFT 00000001 MSFT 00000001)
[    0.071109] ACPI: APIC 0x000000003FFF0300 000452 (v01 VRTUAL MICROSFT 06001702 MSFT 00000097)
[    0.075184] ACPI: OEMB 0x000000003FFFF040 000064 (v01 VRTUAL MICROSFT 06001702 MSFT 00000097)
[    0.079460] SRAT: PXM 0 -> APIC 0x00 -> Node 0
[    0.081643] SRAT: PXM 0 -> APIC 0x01 -> Node 0
[    0.083847] ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0x3fffffff] hotplug
[    0.087160] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x27fffffff] hotplug
[    0.090457] ACPI: SRAT: Node 0 PXM 0 [mem 0x280200000-0xfdfffffff] hotplug
[    0.093834] ACPI: SRAT: Node 0 PXM 0 [mem 0x1000000000-0xffffffffff] hotplug
[    0.097266] ACPI: SRAT: Node 0 PXM 0 [mem 0x10000200000-0x1ffffffffff] hotplug
[    0.100827] ACPI: SRAT: Node 0 PXM 0 [mem 0x20000200000-0x3ffffffffff] hotplug
[    0.109180] NODE_DATA(0) allocated [mem 0x34f32000-0x34f5cfff]
[    0.112114] Zone ranges:
[    0.113485]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.116461]   DMA32    [mem 0x0000000001000000-0x0000000034ffffff]
[    0.119510]   Normal   empty
[    0.120892]   Device   empty
[    0.122274] Movable zone start for each node
[    0.124384] Early memory node ranges
[    0.126361]   node   0: [mem 0x0000000000001000-0x000000000009efff]
[    0.129406]   node   0: [mem 0x0000000029000000-0x0000000034f5cfff]
[    0.133029] Zeroed struct page in unavailable ranges: 36869 pages
[    0.133031] Initmem setup node 0 [mem 0x0000000000001000-0x0000000034f5cfff]
[    0.143898] ACPI: PM-Timer IO Port: 0x408
[    0.145927] APIC: NR_CPUS/possible_cpus limit of 1 almost reached. Keeping one slot for boot cpu.  Processor 0/0x0 ignored.
[    0.151414] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
[    0.154623] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23
[    0.158081] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.161257] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.164541] Using ACPI (MADT) for SMP configuration information
[    0.167650] smpboot: 128 Processors exceeds NR_CPUS limit of 1
[    0.170659] smpboot: Allowing 1 CPUs, 0 hotplug CPUs
[    0.173280] PM: Registered nosave memory: [mem 0x00000000-0x00000fff]
[    0.176584] PM: Registered nosave memory: [mem 0x0009f000-0x28ffffff]
[    0.179894] PM: Registered nosave memory: [mem 0x34f5d000-0x34ffffff]
[    0.184693] [mem 0x40000000-0xffffffff] available for PCI devices
[    0.212824] Booting paravirtualized kernel on bare hardware
[    0.215671] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
[    0.221827] random: get_random_bytes called from start_kernel+0x93/0x503 with crng_init=0
[    0.226026] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1
[    0.230128] percpu: Embedded 45 pages/cpu s147456 r8192 d28672 u2097152
[    0.233727] Hyper-V: PV spinlocks enabled
[    0.235483] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes)
[    0.238597] Built 1 zonelists, mobility grouping on.  Total pages: 48357
[    0.241718] Policy zone: DMA32
[    0.243155] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.0.0-1018-azure root=UUID=7b4b88a7-6d85-44eb-9155-3132fd3d1cd8 ro console=tty1 console=ttyS0 earlyprintk=ttyS0 rootdelay=300 nr_cpus=1 systemd.unit=kdump-tools-dump.service irqpoll nousb ata_piix.prefer_ms_hyperv=0 elfcorehdr=867700K
[    0.257201] Misrouted IRQ fixup and polling support enabled
[    0.259980] This may significantly impact system performance
[    0.272336] Memory: 82316K/196588K available (12292K kernel code, 2232K rwdata, 3840K rodata, 2352K init, 5536K bss, 114272K reserved, 0K cma-reserved)
[    0.279360] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.282719] Kernel/User page tables isolation: enabled
[    0.285246] ftrace: allocating 38598 entries in 151 pages
[    0.303379] rcu: Hierarchical RCU implementation.
[    0.305670] rcu:     RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=1.
[    0.308863]  Tasks RCU enabled.
[    0.310334] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.314124] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
[    0.320349] NR_IRQS: 524544, nr_irqs: 256, preallocated irqs: 16
[    0.323901] Spurious LAPIC timer interrupt on cpu 0
[    0.330364] Console: colour VGA+ 80x25
[    0.000000] Linux version 5.0.0-1018-azure (buildd@lgw01-amd64-048) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) #19~18.04.1-Ubuntu SMP Wed Aug 21 05:13:05 UTC 2019 (Ubuntu 5.0.0-1018.19~18.04.1-azure 5.0.21)
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.0.0-1018-azure root=UUID=7b4b88a7-6d85-44eb-9155-3132fd3d1cd8 ro console=tty1 console=ttyS0 earlyprintk=ttyS0 rootdelay=300 nr_cpus=1 systemd.unit=kdump-tools-dump.service irqpoll nousb ata_piix.prefer_ms_hyperv=0 elfcorehdr=867700K
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000]   AMD AuthenticAMD
[    0.000000]   Hygon HygonGenuine
[    0.000000]   Centaur CentaurHauls
[    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000001000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x0000000029000000-0x0000000034f5cfff] usable
[    0.000000] BIOS-e820: [mem 0x0000000034fffc00-0x0000000034ffffff] usable
[    0.000000] BIOS-e820: [mem 0x000000003fff0000-0x000000003fffefff] ACPI data
[    0.000000] BIOS-e820: [mem 0x000000003ffff000-0x000000003fffffff] ACPI NVS
[    0.000000] printk: bootconsole [earlyser0] enabled
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 2.3 present.
[    0.000000] DMI: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090007  06/02/2017
[    0.000000] Hypervisor detected: Microsoft Hyper-V
[    0.000000] Hyper-V: features 0x2e7f, hints 0x40c2c
[    0.000000] Hyper-V Host Build:14393-10.0-0-0.299
[    0.000000] Hyper-V: LAPIC Timer Frequency: 0xc3500
[    0.000000] tsc: Marking TSC unstable due to running on Hyper-V
[    0.000000] Hyper-V: Using hypercall for remote TLB flush
[    0.000000] tsc: Detected 2394.455 MHz processor
[    0.001399] last_pfn = 0x35000 max_arch_pfn = 0x400000000
[    0.004043] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT
[    0.024144] found SMP MP-table at [mem 0x000ff780-0x000ff78f]
[    0.026977] check: Scanning 1 areas for low memory corruption
[    0.029761] Using GB pages for direct mapping
[    0.032036] RAMDISK: [mem 0x32625000-0x331fffff]
[    0.034267] ACPI: Early table checksum verification disabled
[    0.037090] ACPI: RSDP 0x00000000000F5BF0 000014 (v00 ACPIAM)
[    0.039763] ACPI: RSDT 0x000000003FFF0000 000040 (v01 VRTUAL MICROSFT 06001702 MSFT 00000097)
[    0.043785] ACPI: FACP 0x000000003FFF0200 000081 (v02 VRTUAL MICROSFT 06001702 MSFT 00000097)
[    0.048319] ACPI: DSDT 0x000000003FFF1D24 003CBE (v01 MSFTVM MSFTVM02 00000002 INTL 02002026)
[    0.052474] ACPI: FACS 0x000000003FFFF000 000040
[    0.054736] ACPI: WAET 0x000000003FFF1A80 000028 (v01 VRTUAL MICROSFT 06001702 MSFT 00000097)
[    0.058789] ACPI: SLIC 0x000000003FFF1AC0 000176 (v01 VRTUAL MICROSFT 06001702 MSFT 00000097)
[    0.062902] ACPI: OEM0 0x000000003FFF1CC0 000064 (v01 VRTUAL MICROSFT 06001702 MSFT 00000097)
[    0.066950] ACPI: SRAT 0x000000003FFF0800 000140 (v02 VRTUAL MICROSFT 00000001 MSFT 00000001)
[    0.071109] ACPI: APIC 0x000000003FFF0300 000452 (v01 VRTUAL MICROSFT 06001702 MSFT 00000097)
[    0.075184] ACPI: OEMB 0x000000003FFFF040 000064 (v01 VRTUAL MICROSFT 06001702 MSFT 00000097)

** Affects: lttng-modules (Ubuntu)
     Importance: Undecided
         Status: New

-- 
lttng causes crash with 5.0 based kernel
https://bugs.launchpad.net/bugs/1846485
You received this bug notification because you are a member of Ubuntu LTTng, which is subscribed to lttng-modules in Ubuntu.