← Back to team overview

canonical-ubuntu-qa team mailing list archive

  1. canonical-ubuntu-qa team
  2. Mailing list archive
  3. Message #07864

[Bug 2121673] [NEW] noble ubuntu_ftrace_smoke_test:mmiotrace timeout on aws:r5.metal

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

This happens for 6.8.0-80.80 (2025.08.11) generic kernel and only
happens with aws:r5.metal instance. 6.12 kernel works find. Juerg found
the offending commit to be:

    memcg: drain obj stock on cpu hotplug teardown
    
    BugLink: https://bugs.launchpad.net/bugs/2119458
    
    commit 9f01b4954490d4ccdbcc2b9be34a9921ceee9cbb upstream.
    
    Currently on cpu hotplug teardown, only memcg stock is drained but we
    need to drain the obj stock as well otherwise we will miss the stats
    accumulated on the target cpu as well as the nr_bytes cached. The stats
    include MEMCG_KMEM, NR_SLAB_RECLAIMABLE_B & NR_SLAB_UNRECLAIMABLE_B. In
    addition we are leaking reference to struct obj_cgroup object.
 

Because nothing in the upstream patchset depends on this commit we
decided to delay applying this patch until the next SRU cycle.


INFO | START ubuntu_ftrace_smoke_test.ftrace-smoke-test ubuntu_ftrace_smoke_test.ftrace-smoke-test timeout=900 timestamp=1756180477 localtime=Aug 26 03:54:37
DEBUG| Persistent state client._record_indent now set to 2
DEBUG| Persistent state client.unexpected_reboot now set to ('ubuntu_ftrace_smoke_test.ftrace-smoke-test', 'ubuntu_ftrace_smoke_test.ftrace-smoke-test')
DEBUG| Waiting for pid 3906 for 900 seconds
WARNI| System python is too old, crash handling disabled
DEBUG| Running '/home/ubuntu/autotest/client/tests/ubuntu_ftrace_smoke_test/ubuntu_ftrace_smoke_test.sh'
DEBUG| [stdout] PASSED (CONFIG_FUNCTION_TRACER=y in /boot/config-6.8.0-80-generic)
DEBUG| [stdout] PASSED (CONFIG_FUNCTION_GRAPH_TRACER=y in /boot/config-6.8.0-80-generic)
DEBUG| [stdout] PASSED (CONFIG_STACK_TRACER=y in /boot/config-6.8.0-80-generic)
DEBUG| [stdout] PASSED (CONFIG_DYNAMIC_FTRACE=y in /boot/config-6.8.0-80-generic)
DEBUG| [stdout] PASSED all expected /sys/kernel/debug/tracing files exist
DEBUG| [stdout] PASSED (function_graph in /sys/kernel/debug/tracing/available_tracers)
DEBUG| [stdout] PASSED (function in /sys/kernel/debug/tracing/available_tracers)
DEBUG| [stdout] PASSED (nop in /sys/kernel/debug/tracing/available_tracers)
DEBUG| [stdout] PASSED (tracer function can be enabled)
DEBUG| [stdout] PASSED (tracer function_graph can be enabled)
ERROR| [stderr] grep: /tmp/ftrace-kernel-trace-3910.tmp.log: binary file matches
DEBUG| [stdout] - tracer function_graph got enough data
DEBUG| [stdout] - tracer function_graph completed
DEBUG| [stdout] - tracer function_graph being turned off
ERROR| [stderr] grep: /tmp/ftrace-kernel-trace-3910.tmp.log: binary file matches
DEBUG| [stdout] - tracer got 231 irq events
DEBUG| [stdout] - tracer timerlat got enough data
DEBUG| [stdout] - tracer timerlat completed
DEBUG| [stdout] - tracer timerlat being turned off
DEBUG| [stdout] - tracer nop being set as current tracer
DEBUG| [stdout] PASSED (tracer timerlat can be enabled (got 660 lines of tracing output))
DEBUG| [stdout] - tracer osnoise got enough data
DEBUG| [stdout] - tracer osnoise completed
DEBUG| [stdout] - tracer osnoise being turned off
DEBUG| [stdout] - tracer nop being set as current tracer
DEBUG| [stdout] PASSED (tracer osnoise can be enabled (got 11 lines of tracing output))
DEBUG| [stdout] - tracer hwlat got enough data
DEBUG| [stdout] - tracer hwlat completed
DEBUG| [stdout] - tracer hwlat being turned off
DEBUG| [stdout] - tracer nop being set as current tracer
DEBUG| [stdout] PASSED (tracer hwlat can be enabled (got 13 lines of tracing output))
DEBUG| [stdout] - tracer blk got enough data
DEBUG| [stdout] - tracer blk completed
DEBUG| [stdout] - tracer blk being turned off
DEBUG| [stdout] - tracer nop being set as current tracer
DEBUG| [stdout] PASSED (tracer blk can be enabled (got 2 lines of tracing output))
DEBUG| [stdout] TIMER END Tue Aug 26 03:58:59 UTC 2025
DEBUG| [stdout] TIMEOUT
DEBUG| [stdout] FAILED: aborting, timeout, took way too long to complete
INFO | Timer expired (900 sec.), nuking pid 3906
INFO | ERROR ubuntu_ftrace_smoke_test.ftrace-smoke-test ubuntu_ftrace_smoke_test.ftrace-smoke-test timestamp=1756181377 localtime=Aug 26 04:09:37 Test timeout expired, rc=15
INFO | END ERROR ubuntu_ftrace_smoke_test.ftrace-smoke-test ubuntu_ftrace_smoke_test.ftrace-smoke-test timestamp=1756181377 localtime=Aug 26 04:09:37

** Affects: linux (Ubuntu)
     Importance: Undecided
         Status: New

** Affects: linux (Ubuntu Noble)
     Importance: Undecided
         Status: New

** Also affects: linux (Ubuntu)
   Importance: Undecided
       Status: New

** No longer affects: ubuntu-kernel-tests

** Also affects: linux (Ubuntu Noble)
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Canonical
Platform QA Team, which is subscribed to ubuntu-kernel-tests.
https://bugs.launchpad.net/bugs/2121673

Title:
  noble ubuntu_ftrace_smoke_test:mmiotrace timeout on aws:r5.metal

Status in linux package in Ubuntu:
  New
Status in linux source package in Noble:
  New

Bug description:
  This happens for 6.8.0-80.80 (2025.08.11) generic kernel and only
  happens with aws:r5.metal instance. 6.12 kernel works find. Juerg
  found the offending commit to be:

      memcg: drain obj stock on cpu hotplug teardown
      
      BugLink: https://bugs.launchpad.net/bugs/2119458
      
      commit 9f01b4954490d4ccdbcc2b9be34a9921ceee9cbb upstream.
      
      Currently on cpu hotplug teardown, only memcg stock is drained but we
      need to drain the obj stock as well otherwise we will miss the stats
      accumulated on the target cpu as well as the nr_bytes cached. The stats
      include MEMCG_KMEM, NR_SLAB_RECLAIMABLE_B & NR_SLAB_UNRECLAIMABLE_B. In
      addition we are leaking reference to struct obj_cgroup object.
   

  Because nothing in the upstream patchset depends on this commit we
  decided to delay applying this patch until the next SRU cycle.

  
  INFO | START ubuntu_ftrace_smoke_test.ftrace-smoke-test ubuntu_ftrace_smoke_test.ftrace-smoke-test timeout=900 timestamp=1756180477 localtime=Aug 26 03:54:37
  DEBUG| Persistent state client._record_indent now set to 2
  DEBUG| Persistent state client.unexpected_reboot now set to ('ubuntu_ftrace_smoke_test.ftrace-smoke-test', 'ubuntu_ftrace_smoke_test.ftrace-smoke-test')
  DEBUG| Waiting for pid 3906 for 900 seconds
  WARNI| System python is too old, crash handling disabled
  DEBUG| Running '/home/ubuntu/autotest/client/tests/ubuntu_ftrace_smoke_test/ubuntu_ftrace_smoke_test.sh'
  DEBUG| [stdout] PASSED (CONFIG_FUNCTION_TRACER=y in /boot/config-6.8.0-80-generic)
  DEBUG| [stdout] PASSED (CONFIG_FUNCTION_GRAPH_TRACER=y in /boot/config-6.8.0-80-generic)
  DEBUG| [stdout] PASSED (CONFIG_STACK_TRACER=y in /boot/config-6.8.0-80-generic)
  DEBUG| [stdout] PASSED (CONFIG_DYNAMIC_FTRACE=y in /boot/config-6.8.0-80-generic)
  DEBUG| [stdout] PASSED all expected /sys/kernel/debug/tracing files exist
  DEBUG| [stdout] PASSED (function_graph in /sys/kernel/debug/tracing/available_tracers)
  DEBUG| [stdout] PASSED (function in /sys/kernel/debug/tracing/available_tracers)
  DEBUG| [stdout] PASSED (nop in /sys/kernel/debug/tracing/available_tracers)
  DEBUG| [stdout] PASSED (tracer function can be enabled)
  DEBUG| [stdout] PASSED (tracer function_graph can be enabled)
  ERROR| [stderr] grep: /tmp/ftrace-kernel-trace-3910.tmp.log: binary file matches
  DEBUG| [stdout] - tracer function_graph got enough data
  DEBUG| [stdout] - tracer function_graph completed
  DEBUG| [stdout] - tracer function_graph being turned off
  ERROR| [stderr] grep: /tmp/ftrace-kernel-trace-3910.tmp.log: binary file matches
  DEBUG| [stdout] - tracer got 231 irq events
  DEBUG| [stdout] - tracer timerlat got enough data
  DEBUG| [stdout] - tracer timerlat completed
  DEBUG| [stdout] - tracer timerlat being turned off
  DEBUG| [stdout] - tracer nop being set as current tracer
  DEBUG| [stdout] PASSED (tracer timerlat can be enabled (got 660 lines of tracing output))
  DEBUG| [stdout] - tracer osnoise got enough data
  DEBUG| [stdout] - tracer osnoise completed
  DEBUG| [stdout] - tracer osnoise being turned off
  DEBUG| [stdout] - tracer nop being set as current tracer
  DEBUG| [stdout] PASSED (tracer osnoise can be enabled (got 11 lines of tracing output))
  DEBUG| [stdout] - tracer hwlat got enough data
  DEBUG| [stdout] - tracer hwlat completed
  DEBUG| [stdout] - tracer hwlat being turned off
  DEBUG| [stdout] - tracer nop being set as current tracer
  DEBUG| [stdout] PASSED (tracer hwlat can be enabled (got 13 lines of tracing output))
  DEBUG| [stdout] - tracer blk got enough data
  DEBUG| [stdout] - tracer blk completed
  DEBUG| [stdout] - tracer blk being turned off
  DEBUG| [stdout] - tracer nop being set as current tracer
  DEBUG| [stdout] PASSED (tracer blk can be enabled (got 2 lines of tracing output))
  DEBUG| [stdout] TIMER END Tue Aug 26 03:58:59 UTC 2025
  DEBUG| [stdout] TIMEOUT
  DEBUG| [stdout] FAILED: aborting, timeout, took way too long to complete
  INFO | Timer expired (900 sec.), nuking pid 3906
  INFO | ERROR ubuntu_ftrace_smoke_test.ftrace-smoke-test ubuntu_ftrace_smoke_test.ftrace-smoke-test timestamp=1756181377 localtime=Aug 26 04:09:37 Test timeout expired, rc=15
  INFO | END ERROR ubuntu_ftrace_smoke_test.ftrace-smoke-test ubuntu_ftrace_smoke_test.ftrace-smoke-test timestamp=1756181377 localtime=Aug 26 04:09:37

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2121673/+subscriptions
  Thread Previous • Date Previous • • Thread Next