kernel-packages team mailing list archive

Thread
Date
[Bug 1403459] Re: kernel bug when hotplugging a CPU

To: kernel-packages@xxxxxxxxxxxxxxxxxxx
From: Bug Watch Updater <1403459@xxxxxxxxxxxxxxxxxx>
Date: Wed, 17 Dec 2014 11:28:31 -0000
Reply-to: Bug 1403459 <1403459@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Launchpad has imported 8 comments from the remote bug at
https://bugzilla.novell.com/show_bug.cgi?id=908794.

If you reply to an imported comment from within Launchpad, your comment
will be sent to the remote bug automatically. Read more about
Launchpad's inter-bugtracker facilities at
https://help.launchpad.net/InterBugTracking.

------------------------------------------------------------------------
On 2014-12-08T09:58:29+00:00 Benjamin Drung wrote:

When I hot-plug a CPU to an openSUSE VM running on KVM, I get following
kernel oops:

# uname -a
Linux opensuse 3.16.6-2-default #1 SMP Mon Oct 20 13:47:22 UTC 2014 (feb42ea) x86_64 x86_64 x86_64 GNU/Linux
# dmesg
[   54.927758] CPU2 has been hot-added
[   54.928610] smpboot: Booting Node 0 Processor 2 APIC 0x2
[    0.004000] kvm-clock: cpu 2, msr 2:403de081, secondary cpu clock
[   54.940001] TSC synchronization [CPU#1 -> CPU#2]:
[   54.940001] Measured 195274448514 cycles TSC warp between CPUs, turning off TSC clock.
[   54.940001] tsc: Marking TSC unstable due to check_tsc_sync_source failed
[   54.960222] KVM setup async PF for cpu 2
[   54.960265] kvm-stealtime: cpu 2, msr 24008d240
[   54.960342] microcode: CPU2 sig=0x306c1, pf=0x1, revision=0x1
[   55.012464] Will online and init hotplugged CPU: 2
[   55.012574] kobject '(null)' (ffff88024009a3d8): tried to add an uninitialized object, something is seriously wrong.
[   55.012615] CPU: 1 PID: 892 Comm: systemd-udevd Not tainted 3.16.6-2-default #1
[   55.012649] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_171129-lamiak 04/01/2014
[   55.012688]  ffff8802347cbd98 ffffffff815ca482 ffff88024009a3d8 ffffffff8130e153
[   55.012729]  ffffffff00000020 ffff8802347cbda8 ffff8802347cbd50 00000001334da7a8
[   55.012766]  000000000000b0a0 0000000001ca6930 0000000000000002 ffff8802334da298
[   55.012804] Call Trace:
[   55.012845]  [<ffffffff81005096>] dump_trace+0x86/0x330
[   55.012881]  [<ffffffff810053d4>] show_stack_log_lvl+0x94/0x170
[   55.012919]  [<ffffffff81006a31>] show_stack+0x21/0x50
[   55.012955]  [<ffffffff815ca482>] dump_stack+0x41/0x51
[   55.012990]  [<ffffffff8130e153>] kobject_add+0xa3/0xb0
[   55.013027]  [<ffffffff812ed4a1>] blk_mq_register_hctx+0x91/0xb0
[   55.013062]  [<ffffffff812edd5e>] blk_mq_sysfs_register+0x3e/0x50
[   55.013095]  [<ffffffff812eb9c2>] blk_mq_queue_reinit_notify+0x72/0xb0
[   55.013129]  [<ffffffff810806bc>] notifier_call_chain+0x4c/0x70
[   55.013162]  [<ffffffff8105beef>] cpu_notify+0x1f/0x40
[   55.013195]  [<ffffffff8105c0d7>] _cpu_up+0x157/0x170
[   55.013226]  [<ffffffff8105c169>] cpu_up+0x79/0xa0
[   55.013260]  [<ffffffff815bf555>] cpu_subsys_online+0x35/0x80
[   55.013294]  [<ffffffff8140c64d>] device_online+0x5d/0xa0
[   55.013328]  [<ffffffff8140c705>] online_store+0x75/0x80
[   55.013360]  [<ffffffff8122127a>] kernfs_fop_write+0xda/0x150
[   55.013394]  [<ffffffff811b1aa2>] vfs_write+0xb2/0x1f0
[   55.013426]  [<ffffffff811b25e2>] SyS_write+0x42/0xa0
[   55.013462]  [<ffffffff815d0c2d>] system_call_fastpath+0x1a/0x1f
[   55.013517]  [<00007f9fbf3804e0>] 0x7f9fbf3804df

I can reproduce it on a Ubuntu 14.10 host with stock KVM and on Debian
host with a custom KVM. The oops sometimes causes the virtual machine to
freeze.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1403459/comments/0

------------------------------------------------------------------------
On 2014-12-08T12:43:55+00:00 Tiwai-r wrote:

Could you check whether the issue is reproduced on 3.17 and 3.18 kernels
available at OBS Kernel:stable and Kernel:HEAD repos?

Reply at:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1403459/comments/1

------------------------------------------------------------------------
On 2014-12-08T13:03:42+00:00 Benjamin Drung wrote:

(In reply to Takashi Iwai from comment #1)
> Could you check whether the issue is reproduced on 3.17 and 3.18 kernels
> available at OBS Kernel:stable and Kernel:HEAD repos?

Same result with linux 3.17 from OBS Kernel:stable:

# uname -a
Linux opensuse 3.17.4-4.g8622a2e-default #1 SMP Sat Dec 6 20:43:41 UTC 2014 (8622a2e) x86_64 x86_64 x86_64 GNU/Linux
# dmesg
[  117.297802] CPU1 has been hot-added
[  117.298543] SMP alternatives: switching to SMP code
[  117.306886] x86: Booting SMP configuration:
[  117.306922] smpboot: Booting Node 0 Processor 1 APIC 0x1
[    0.008000] kvm-clock: cpu 1, msr 2:403de041, secondary cpu clock
[  117.320007] TSC synchronization [CPU#0 -> CPU#1]:
[  117.320007] Measured 407089711478 cycles TSC warp between CPUs, turning off TSC clock.
[  117.320007] tsc: Marking TSC unstable due to check_tsc_sync_source failed
[  117.338582] KVM setup async PF for cpu 1
[  117.338626] kvm-stealtime: cpu 1, msr 24004d240
[  117.338727] kobject '(null)' (ffff88024005a558): tried to add an uninitialized object, something is seriously wrong.
[  117.338728] CPU: 0 PID: 841 Comm: systemd-udevd Not tainted 3.17.4-4.g8622a2e-default #1
[  117.338729] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_171129-lamiak 04/01/2014
[  117.338730]  ffff8800bbaf3d98 ffffffff815dc842 ffff88024005a558 ffffffff813181c3
[  117.338731]  ffffffff00000020 ffff8800bbaf3da8 ffff8800bbaf3d50 0000000133a94f28
[  117.338732]  000000000000b0a0 0000000001e779d0 0000000000000001 ffff880233a94a18
[  117.338732] Call Trace:
[  117.338740]  [<ffffffff81005246>] dump_trace+0x86/0x330
[  117.338742]  [<ffffffff81005584>] show_stack_log_lvl+0x94/0x170
[  117.338744]  [<ffffffff81006be1>] show_stack+0x21/0x50
[  117.338746]  [<ffffffff815dc842>] dump_stack+0x41/0x51
[  117.338749]  [<ffffffff813181c3>] kobject_add+0xa3/0xb0
[  117.338753]  [<ffffffff812f74a1>] blk_mq_register_hctx+0x91/0xb0
[  117.338755]  [<ffffffff812f7d5e>] blk_mq_sysfs_register+0x3e/0x50
[  117.338756]  [<ffffffff812f5962>] blk_mq_queue_reinit_notify+0x72/0xb0
[  117.338759]  [<ffffffff8107985c>] notifier_call_chain+0x4c/0x70
[  117.338761]  [<ffffffff8105d6cf>] cpu_notify+0x1f/0x40
[  117.338763]  [<ffffffff8105d8b7>] _cpu_up+0x157/0x170
[  117.338765]  [<ffffffff8105d949>] cpu_up+0x79/0xa0
[  117.338768]  [<ffffffff815d1625>] cpu_subsys_online+0x35/0x80
[  117.338776]  [<ffffffff814150ed>] device_online+0x5d/0xa0
[  117.338779]  [<ffffffff814151a5>] online_store+0x75/0x80
[  117.338782]  [<ffffffff81227c4a>] kernfs_fop_write+0xda/0x150
[  117.338785]  [<ffffffff811b7d62>] vfs_write+0xb2/0x1f0
[  117.338787]  [<ffffffff811b88a2>] SyS_write+0x42/0xa0
[  117.338788]  [<ffffffff815e2f6d>] system_call_fastpath+0x1a/0x1f
[  117.338800]  [<00007f5728fbd4e0>] 0x7f5728fbd4e0
[  117.338826] microcode: CPU1 sig=0x306c1, pf=0x1, revision=0x1
[  117.393791] Will online and init hotplugged CPU: 1

Reply at:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1403459/comments/2

------------------------------------------------------------------------
On 2014-12-08T13:09:34+00:00 Benjamin Drung wrote:

Same result with linux 3.18 from OBS Kernel:HEAD:

# uname -a
Linux opensuse 3.18.0-rc7-2.g088d59b-default #1 SMP Fri Dec 5 00:03:54 UTC 2014 (088d59b) x86_64 x86_64 x86_64 GNU/Linux
# dmesg
[  150.252464] CPU2 has been hot-added
[  150.253350] smpboot: Booting Node 0 Processor 2 APIC 0x2
[    0.004000] kvm-clock: cpu 2, msr 2:403de081, secondary cpu clock
[  150.268003] TSC synchronization [CPU#1 -> CPU#2]:
[  150.268003] Measured 518729842258 cycles TSC warp between CPUs, turning off TSC clock.
[  150.268003] tsc: Marking TSC unstable due to check_tsc_sync_source failed
[  150.285117] KVM setup async PF for cpu 2
[  150.285201] kvm-stealtime: cpu 2, msr 24008c280
[  150.296118] kobject '(null)' (ffffe8ffffc8b5d8): tried to add an uninitialized object, something is seriously wrong.
[  150.296197] CPU: 1 PID: 1386 Comm: systemd-udevd Not tainted 3.18.0-rc7-2.g088d59b-default #1
[  150.296243] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_171129-lamiak 04/01/2014
[  150.296285]  0000000000000000 0000000000000002 ffffffff81605f07 ffffe8ffffc8b5d8
[  150.296332]  ffffffff8132c7a0 ffff88023341d370 0000000000000020 ffff8800bb05bd58
[  150.296382]  ffff8800bb05bd08 000000000000a0a0 000000003f441940 0000000000000007
[  150.296427] Call Trace:
[  150.296479]  [<ffffffff81005306>] dump_trace+0x86/0x330
[  150.296541]  [<ffffffff81005644>] show_stack_log_lvl+0x94/0x170
[  150.296596]  [<ffffffff81006d21>] show_stack+0x21/0x50
[  150.296673]  [<ffffffff81605f07>] dump_stack+0x41/0x51
[  150.296721]  [<ffffffff8132c7a0>] kobject_add+0xa0/0xb0
[  150.296767]  [<ffffffff8130aee1>] blk_mq_register_hctx+0x91/0xb0
[  150.296809]  [<ffffffff8130b82e>] blk_mq_sysfs_register+0x3e/0x60
[  150.296857]  [<ffffffff81309298>] blk_mq_queue_reinit_notify+0xf8/0x190
[  150.296900]  [<ffffffff8107cfdc>] notifier_call_chain+0x4c/0x70
[  150.296947]  [<ffffffff8105fd23>] cpu_notify+0x23/0x50
[  150.296988]  [<ffffffff81060037>] _cpu_up+0x157/0x170
[  150.297033]  [<ffffffff810600d9>] cpu_up+0x89/0xb0
[  150.297079]  [<ffffffff815fa5b5>] cpu_subsys_online+0x35/0x80
[  150.297127]  [<ffffffff814323cd>] device_online+0x5d/0xa0
[  150.297185]  [<ffffffff81432485>] online_store+0x75/0x80
[  150.297238]  [<ffffffff81236a5a>] kernfs_fop_write+0xda/0x150
[  150.297309]  [<ffffffff811c5532>] vfs_write+0xb2/0x1f0
[  150.297367]  [<ffffffff811c5f42>] SyS_write+0x42/0xb0
[  150.297442]  [<ffffffff8160c4ed>] system_call_fastpath+0x16/0x1b
[  150.297493]  [<00007f0132fb24e0>] 0x7f0132fb24e0
[  150.297583] microcode: CPU2 sig=0x306c1, pf=0x1, revision=0x1
[  150.297727] Will online and init hotplugged CPU: 2

Reply at:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1403459/comments/3

------------------------------------------------------------------------
On 2014-12-08T13:11:21+00:00 Tiwai-r wrote:

Thanks.  So the bug still remains in the upstream,  so we need to report
it.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1403459/comments/4

------------------------------------------------------------------------
On 2014-12-10T14:18:25+00:00 Tiwai-r wrote:

OK, I could reproduce this easily, too, and spotted the culprit.
The fix patch is below.  I'm going to send the fix to upstream.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1403459/comments/5

------------------------------------------------------------------------
On 2014-12-10T14:18:49+00:00 Tiwai-r wrote:

Created attachment 616585
Fix patch

Reply at:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1403459/comments/6

------------------------------------------------------------------------
On 2014-12-14T10:27:03+00:00 Tiwai-r wrote:

The patch was merged to Linus tree, and backported to openSUSE 13.2 and
stable branches now.  SLE12 doesn't need this.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1403459/comments/7


** Changed in: linux (openSUSE)
       Status: Unknown => Fix Released

** Changed in: linux (openSUSE)
   Importance: Unknown => Medium

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1403459

Title:
   kernel bug when hotplugging a CPU

Status in linux package in Ubuntu:
  Incomplete
Status in linux package in Debian:
  Unknown
Status in linux package in openSUSE:
  Fix Released

Bug description:
  When I hot-plug a CPU to an Ubuntu 14.10 VM running on KVM, I get
  following kernel bug. Ubuntu 14.04 is not affected by this bug.

  # uname -a
  Linux ubuntu 3.16.0-28-generic #38-Ubuntu SMP Fri Dec 12 17:37:40 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
  # dmesg
  [  109.469094] CPU1 has been hot-added
  [  109.477005] SMP alternatives: switching to SMP code
  [  109.490583] x86: Booting SMP configuration:
  [  109.490586] smpboot: Booting Node 0 Processor 1 APIC 0x1
  [  109.476344] kvm-clock: cpu 1, msr 0:3ffe6041, secondary cpu clock
  [  109.586704] kvm: enabling virtualization on CPU1
  [  109.596330] KVM setup async PF for cpu 1
  [  109.596330] kvm-stealtime: cpu 1, msr 3de2d100
  [  109.596645] microcode: CPU1: patch_level=0x01000065
  [  109.596910] Will online and init hotplugged CPU: 1
  [  109.598469] kobject '(null)' (ffff88003de3b5d8): tried to add an uninitialized object, something is seriously wrong.
  [  109.599255] CPU: 0 PID: 1424 Comm: systemd-udevd Not tainted 3.16.0-28-generic #38-Ubuntu
  [  109.599257] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
  [  109.599259]  ffff880036465250 ffff88003af77c88 ffffffff81781eaa ffff88003de3b5d8
  [  109.599263]  ffff88003af77cf8 ffffffff8139e896 ffff880000000020 ffff88003af77d08
  [  109.599266]  ffff88003af77cb0 ffff88003af77cc8 ffffffff8137967b 0000000000000001
  [  109.599269] Call Trace:
  [  109.599276]  [<ffffffff81781eaa>] dump_stack+0x45/0x56
  [  109.599280]  [<ffffffff8139e896>] kobject_add+0xa6/0xb0
  [  109.599283]  [<ffffffff8137967b>] ? get_first_sibling+0x2b/0x40
  [  109.599287]  [<ffffffff8139d08d>] ? ida_get_new_above+0x1dd/0x210
  [  109.599291]  [<ffffffff81378b66>] blk_mq_register_hctx+0xa6/0xc0
  [  109.599293]  [<ffffffff813794c1>] blk_mq_sysfs_register+0x41/0x60
  [  109.599297]  [<ffffffff81376fc2>] blk_mq_queue_reinit_notify+0x72/0xb0
  [  109.599300]  [<ffffffff8109972d>] notifier_call_chain+0x4d/0x70
  [  109.599303]  [<ffffffff8109983e>] __raw_notifier_call_chain+0xe/0x10
  [  109.599306]  [<ffffffff81070133>] cpu_notify+0x23/0x50
  [  109.599309]  [<ffffffff81070348>] _cpu_up+0x168/0x180
  [  109.599312]  [<ffffffff810703ea>] cpu_up+0x8a/0xb0
  [  109.599315]  [<ffffffff81778ee0>] cpu_subsys_online+0x40/0xa0
  [  109.599319]  [<ffffffff814cf225>] device_online+0x65/0x90
  [  109.599322]  [<ffffffff814cf2cd>] online_store+0x7d/0x90
  [  109.599325]  [<ffffffff814cc658>] dev_attr_store+0x18/0x30
  [  109.599328]  [<ffffffff8125950a>] sysfs_kf_write+0x3a/0x50
  [  109.599331]  [<ffffffff812586ae>] kernfs_fop_write+0xee/0x180
  [  109.599335]  [<ffffffff811e10b7>] vfs_write+0xb7/0x1f0
  [  109.599338]  [<ffffffff811e1c66>] SyS_write+0x46/0xb0
  [  109.599342]  [<ffffffff81789eed>] system_call_fastpath+0x1a/0x1f

  I initially reported this bug in openSUSE and it was fixed in the
  Linus tree:
  https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=06a41a9

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1403459/+subscriptions
References

[Bug 1403459] [NEW] kernel bug when hotplugging a CPU
From: Benjamin Drung, 2014-12-17