kernel-packages team mailing list archive

Thread
Date
[Bug 1354024] [NEW] Running YCSB workload on MongoDB on Ubuntu 14.10 VM resulted in kernel bug

To: kernel-packages@xxxxxxxxxxxxxxxxxxx
From: Launchpad Bug Tracker <1354024@xxxxxxxxxxxxxxxxxx>
Date: Fri, 15 Aug 2014 18:25:24 -0000
Reply-to: Bug 1354024 <1354024@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
You have been subscribed to a public bug:

== Comment: #0 - Kalpana Shetty <kalshett@xxxxxxxxxx> - 2014-08-05 23:53:28 ==
---Problem Description---
Running YCSB workload on LongoDB on Ubuntu 14.10 VM resulted in kernel bug
 
---uname output---
root@u10vm15:~# uname -a Linux u10vm15 3.16.0-6-generic #11-Ubuntu SMP Mon Jul 28 02:00:45 UTC 2014 ppc64le ppc64le ppc64le GNU/Linux
 
---Additional Hardware Info---
Power 8 - Tuleta 

 Machine Type = POWER 8 
 
---System Hang---
 Ubuntu 14.10 LE guest needs to be restarted when seen this issue.
 
 Steps to reproduce:
- Install Ubuntu 14.10 on 2 VMs(July 30th build)
- Run Mongodb 2.6.2 on one of PowerKVM VM 
- Run YCSB 0.1.4 on other VM
- Create 1million record load on MongoDB using YCSB;  allow it to run for 4 to 5 hours or so. 

Setup details:
- MongoDB server on one VM (version: 2.6.2)
- YCSB workload running on one VM (YCSB version - ycsb-0.1.4)

uname on Host:
[root@powerkvm5-lp1 ~]# uname -a
Linux powerkvm5-lp1.austin.ibm.com 3.10.42-2004.pkvm2_1_1.8.ppc64 #1 SMP Fri Jul 18 11:20:03 CDT 2014 ppc64 ppc64 ppc64 GNU/Linux

uname on Guest OS:
root@u10vm15:~# uname -a
Linux u10vm15 3.16.0-6-generic #11-Ubuntu SMP Mon Jul 28 02:00:45 UTC 2014 ppc64le ppc64le ppc64le GNU/Linux

[23001.071911] ------------[ cut here ]------------
[23001.071922] kernel BUG at /build/buildd/linux-3.16.0/fs/dcache.c:1626!
[23001.072917] Oops: Exception in kernel mode, sig: 5 [#1]
[23001.073620] SMP NR_CPUS=2048 NUMA pSeries
[23001.074149] Modules linked in: pseries_rng rtc_generic ohci_pci
[23001.075162] CPU: 8 PID: 3384 Comm: updatedb.mlocat Not tainted 3.16.0-6-generic #11-Ubuntu
[23001.076006] task: c000000006e00000 ti: c000000130364000 task.ti: c000000130364000
[23001.076834] NIP: c0000000002abc68 LR: c0000000002abf90 CTR: c00000000001f880
[23001.077650] REGS: c0000001303676d0 TRAP: 0700   Not tainted  (3.16.0-6-generic)
[23001.078468] MSR: 8000000100029033 <SF,EE,ME,IR,DR,RI,LE>  CR: 24004842  XER: 20000000
[23001.080432] CFAR: c0000000002abf8c SOFTE: 1 
[23001.080432] GPR00: c0000000002abf90 c000000130367950 c000000001346618 c000000005dd0000 
[23001.080432] GPR04: 0000000000000000 0000000000001000 c000000005dcd170 0000000000000fcc 
[23001.080432] GPR08: 0000000000001000 0000000000000001 8803dabf05ffffff 000000000016eb0c 
[23001.080432] GPR12: 0000000000004400 c00000000fe41c00 0000000000000000 0000000000100000 
[23001.080432] GPR16: 000000001001d660 0000010034e94ec0 0000000000000001 0000000053d94034 
[23001.080432] GPR20: 0000000000000000 0000000000000001 00003fffcaa1efb8 0000010034e842e0 
[23001.080432] GPR24: 0000000000000000 0000000000000000 0000010034e94ec0 ffffffffffffff9c 
[23001.080432] GPR28: 0000000000000040 0000000000000000 c000000005dd0000 0000000000000000 
[23001.091266] NIP [c0000000002abc68] d_instantiate+0x38/0xf0
[23001.091837] LR [c0000000002abf90] d_splice_alias+0x60/0x1a0
[23001.092404] Call Trace:
[23001.092692] [c000000130367980] [c0000000002abf90] d_splice_alias+0x60/0x1a0
[23001.093544] [c0000001303679c0] [c00000000034c5b4] ext4_lookup+0xc4/0x1c0
[23001.094399] [c000000130367a50] [c000000000299944] lookup_real+0x64/0xc0
[23001.095261] [c000000130367a90] [c00000000029a790] __lookup_hash+0x60/0x80
[23001.096106] [c000000130367ae0] [c00000000029d610] lookup_slow+0x70/0x110
[23001.096946] [c000000130367b20] [c00000000029ea08] path_lookupat+0x958/0x9a0
[23001.097804] [c000000130367be0] [c00000000029eaa8] filename_lookup+0x58/0x140
[23001.098648] [c000000130367c30] [c0000000002a2524] user_path_at_empty+0x84/0xe0
[23001.099580] [c000000130367d20] [c0000000002937e4] vfs_fstatat+0x84/0x140
[23001.100432] [c000000130367d80] [c000000000293eb4] SyS_newlstat+0x34/0x60
[23001.101378] [c000000130367e30] [c00000000000a0fc] syscall_exit+0x0/0x7c
[23001.102193] Instruction dump:
[23001.102589] 7c0802a6 fbc1fff0 fbe1fff8 f8010010 f821ffd1 7c7e1b78 7c9f2378 60000000 
[23001.103945] 60000000 e93e00b8 3149ffff 7d2a4910 <0b090000> 2fbf0000 419e0060 387f0088 
[23001.105276] ---[ end trace b20dd6fbb5b21932 ]---
[23001.118598] 
root@u10vm15:~# 

After I rebooted I'm keep seeing below call traces:
Ubuntu Utopic Unicorn (development branch) u10vm15 hvc0

u10vm15 login: root
Password: 
Last login: Wed Aug  6 00:02:18 IST 2014 on hvc0
Welcome to Ubuntu Utopic Unicorn (development branch) (GNU/Linux 3.16.0-6-generic ppc64le)

 * Documentation:  https://help.ubuntu.com/
[32950.678160] systemd-logind[1071]: Removed session c1.
[32950.694697] systemd-logind[1071]: New session c2 of user root.
[32950.703411] Unable to handle kernel paging request for data at address 0x2f0000000000000
[32950.704886] Faulting instruction address: 0xc000000000260290
[32950.706148] Oops: Kernel access of bad area, sig: 11 [#2]
[32950.707098] SMP NR_CPUS=2048 NUMA pSeries
[32950.708098] Modules linked in: pseries_rng rtc_generic ohci_pci
[32950.709651] CPU: 8 PID: 342 Comm: cgmanager Tainted: G      D       3.16.0-6-generic #11-Ubuntu
[32950.711433] task: c00000012e3854c0 ti: c00000012e410000 task.ti: c00000012e410000
[32950.712938] NIP: c000000000260290 LR: c000000000260384 CTR: c0000000004099f0
[32950.715490] REGS: c00000012e413970 TRAP: 0300   Tainted: G      D        (3.16.0-6-generic)
[32950.718098] MSR: 8000000100009033 <SF,EE,ME,IR,DR,RI,LE>  CR: 28002448  XER: 00000000
[32950.723688] CFAR: c000000000010a30 DAR: 02f0000000000000 DSISR: 40000000 SOFTE: 1 
GPR00: c000000000260384 c00000012e413bf0 c000000001346618 0000000000000000 
GPR04: 00000000000000d0 c0000000012a4ad0 0000000000000001 c000000001919db0 
GPR08: 0000000000000e90 0000000000000000 0000000000b30000 00065581e050bfe3 
GPR12: c0000000004099f0 c00000000fe41c00 fffffffffffffe80 fffffffffffffe90 
GPR16: fffffffffffffea0 fffffffffffffeb0 fffffffffffffec0 fffffffffffffed0 
GPR20: fffffffffffffee0 fffffffffffffef0 ffffffffffffff00 ffffffffffffff10 
GPR24: ffffffffffffff20 00003ffff2b48de0 c00000013604c600 00003ffff2b48b18 
GPR28: c0000000002adc8c 00000000000000d0 02f0000000000000 c00000013604c600 
[32950.742460] NIP [c000000000260290] kmem_cache_alloc+0x90/0x2d0
[32950.743997] LR [c000000000260384] kmem_cache_alloc+0x184/0x2d0
[32950.745105] Call Trace:
[32950.745637] [c00000012e413bf0] [c000000000260384] kmem_cache_alloc+0x184/0x2d0 (unreliable)
[32950.747609] [c00000012e413c40] [c0000000002adc8c] __d_alloc+0x4c/0x1c0
[32950.749004] [c00000012e413c80] [c00000000083bd58] sock_alloc_file+0x78/0x170
[32950.750433] [c00000012e413ce0] [c000000000841244] SyS_accept4+0xd4/0x280
[32950.751833] [c00000012e413dc0] [c000000000842c50] SyS_socketcall+0x3c0/0x400
[32950.753239] [c00000012e413e30] [c00000000000a0fc] syscall_exit+0x0/0x7c
[32950.754577] Instruction dump:
[32950.755324] 7f5fd378 e94d0040 e93f0000 7ce95214 e9070008 7fc9502a e9270010 2fbe0000 
[32950.757463] 41de0070 2fa90000 419e0068 e93f0022 <7f7e482a> 39200000 88cd02ba 992d02ba 
[32950.759787] ---[ end trace b20dd6fbb5b21933 ]---
[32950.775708] 
[32950.813539] systemd-logind[1071]: cgmanager: Error pinging manager: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
[32950.821827] systemd-logind[1071]: Failed to create cpuset:/user/0.user/c2.session: No such file or directory
[32950.825834] init: cgmanager main process (342) killed by SEGV signal
[32950.828432] systemd-logind[1071]: Failed to create devices:/user/0.user/c2.session: No such file or directory
[32950.832066] init: cgmanager main process ended, respawning
[32950.833739] systemd-logind[1071]: Failed to create freezer:/user/0.user/c2.session: No such file or directory
[32950.836759] systemd-logind[1071]: Failed to create hugetlb:/user/0.user/c2.session: No such file or directory
[32950.839464] systemd-logind[1071]: Failed to create memory:/user/0.user/c2.session: No such file or directory
[32950.842165] systemd-logind[1071]: Failed to create perf_event:/user/0.user/c2.session: No such file or directory
[32950.844964] systemd-logind[1071]: Failed to create net_cls:/user/0.user/c2.session: No such file or directory
[32950.847865] systemd-logind[1071]: Failed to create net_prio:/user/0.user/c2.session: No such file or directory
[32950.862660] Unable to handle kernel paging request for data at address 0x2f0000000000000
[32950.864661] Faulting instruction address: 0xc000000000260290
[32950.866115] Oops: Kernel access of bad area, sig: 11 [#3]
[32950.867269] SMP NR_CPUS=2048 NUMA pSeries
[32950.868437] Modules linked in: pseries_rng rtc_generic ohci_pci
[32950.870457] CPU: 8 PID: 3550 Comm: sh Tainted: G      D       3.16.0-6-generic #11-Ubuntu
[32950.872181] task: c00000012d70e910 ti: c00000012d7a0000 task.ti: c00000012d7a0000
[32950.873907] NIP: c000000000260290 LR: c000000000260384 CTR: c000000000409c60
[32950.875637] REGS: c00000012d7a3780 TRAP: 0300   Tainted: G      D        (3.16.0-6-generic)
[32950.877373] MSR: 8000000000009033 <SF,EE,ME,IR,DR,RI,LE>  CR: 22002888  XER: 20000000
[32950.881408] CFAR: c00000000006aec8 DAR: 02f0000000000000 DSISR: 40000000 SOFTE: 1 
GPR00: c000000000260384 c00000012d7a3a00 c000000001346618 0000000000000000 
GPR04: 00000000000000d0 0000000000000001 c00000012d7a3b40 c000000001919db0 
GPR08: 0000000000000e90 0000000000000000 0000000000b30000 c000000007c8a02d 
GPR12: 0000000000002200 c00000000fe41c00 0000000000000000 0000000056fbd0b8 
GPR16: 0000000056fbffa8 0000000056fbfeb8 0000010013380328 00003fffc4b6fed6 
GPR20: 0000000000000000 0000010013380340 0000000056fbfe60 0000000000000004 
GPR24: c00000012eeca100 c00000012d538300 c00000013604c600 0000000000000001 
GPR28: c0000000002adc8c 00000000000000d0 02f0000000000000 c00000013604c600 
[32950.904699] NIP [c000000000260290] kmem_cache_alloc+0x90/0x2d0
[32950.905897] LR [c000000000260384] kmem_cache_alloc+0x184/0x2d0
[32950.907011] Call Trace:
[32950.907437] [c00000012d7a3a00] [c000000000260384] kmem_cache_alloc+0x184/0x2d0 (unreliable)
[32950.909026] [c00000012d7a3a50] [c0000000002adc8c] __d_alloc+0x4c/0x1c0
[32950.910130] [c00000012d7a3a90] [c0000000002ade38] d_alloc+0x38/0xd0
[32950.911170] [c00000012d7a3ad0] [c00000000029a6fc] lookup_dcache+0x10c/0x140
[32950.912159] [c00000012d7a3b20] [c00000000029a774] __lookup_hash+0x44/0x80
[32950.913141] [c00000012d7a3b70] [c00000000029d610] lookup_slow+0x70/0x110
[32950.914120] [c00000012d7a3bb0] [c00000000029ea08] path_lookupat+0x958/0x9a0
[32950.915095] [c00000012d7a3c70] [c00000000029eaa8] filename_lookup+0x58/0x140
[32950.916069] [c00000012d7a3cc0] [c0000000002a2524] user_path_at_empty+0x84/0xe0
[32950.917208] [c00000012d7a3db0] [c000000000289908] SyS_faccessat+0xc8/0x2f0
[32950.918183] [c00000012d7a3e30] [c00000000000a0fc] syscall_exit+0x0/0x7c
[32950.919157] Instruction dump:
[32950.919643] 7f5fd378 e94d0040 e93f0000 7ce95214 e9070008 7fc9502a e9270010 2fbe0000 
[32950.921265] 41de0070 2fa90000 419e0068 e93f0022 <7f7e482a> 39200000 88cd02ba 992d02ba 
[32950.923055] ---[ end trace b20dd6fbb5b21934 ]---
[32950.939809] 
[32950.940617] init: cgmanager main process (3550) killed by SEGV signal
[32950.942078] init: cgmanager main process ended, respawning
root@u10vm15:~# [32955.627058] Unable to handle kernel paging request for data at address 0x82cf8c206002008
[32955.628340] Faulting instruction address: 0xc00000000035eca8
[32955.629471] Oops: Kernel access of bad area, sig: 11 [#4]
[32955.630336] SMP NR_CPUS=2048 NUMA pSeries
[32955.631340] Modules linked in: pseries_rng rtc_generic ohci_pci
[32955.633486] CPU: 1 PID: 217 Comm: jbd2/sda2-8 Tainted: G      D       3.16.0-6-generic #11-Ubuntu
[32955.635062] task: c00000012d983f90 ti: c00000012da0c000 task.ti: c00000012da0c000
[32955.636627] NIP: c00000000035eca8 LR: c00000000035ec84 CTR: c00000000035ec00
[32955.638053] REGS: c00000012da0f7e0 TRAP: 0300   Tainted: G      D        (3.16.0-6-generic)
[32955.639479] MSR: 8000000100009033 <SF,EE,ME,IR,DR,RI,LE>  CR: 24022048  XER: 00000000
[32955.642493] CFAR: c00000000013bd44 DAR: 082cf8c206002008 DSISR: 42000000 SOFTE: 1 
GPR00: c00000000035ec84 c00000012da0fa60 c000000001346618 c00000012d4c5a88 
GPR04: c000000005de0000 0000000001f3a8b9 0030fc0000000000 00001df91effac68 
GPR08: 0000000000000ccd 082bb8eb06002000 082cf8c206002000 00001b76fc8cf2f7 
GPR12: c00000000035ec00 c00000000fe40380 6db6db6db6db6db7 0000000000000000 
GPR16: 0000000000000000 0000000000000001 0000000000080000 0000000000000020 
GPR20: c00000012fd6e824 0000000000000040 0000000000000008 0000000000400000 
GPR24: 0000000000000000 0000000000000000 0000000000000000 c00000012d4c4800 
GPR28: c00000012d4c5a88 c0000000052b4cd0 c00000012d4c5800 c0000000052b4c00 
[32955.654852] NIP [c00000000035eca8] ext4_journal_commit_callback+0xa8/0x170
[32955.655489] LR [c00000000035ec84] ext4_journal_commit_callback+0x84/0x170
[32955.656137] Call Trace:
[32955.656382] [c00000012da0fa60] [c00000000035ec84] ext4_journal_commit_callback+0x84/0x170 (unreliable)
[32955.657670] [c00000012da0fac0] [c00000000039bb5c] jbd2_journal_commit_transaction+0x171c/0x1ea0
[32955.658712] [c00000012da0fcf0] [c0000000003a348c] kjournald2+0xec/0x300
[32955.659516] [c00000012da0fd80] [c0000000000cbc30] kthread+0x110/0x130
[32955.660317] [c00000012da0fe30] [c00000000000a3e8] ret_from_kernel_thread+0x5c/0x74
[32955.661207] Instruction dump:
[32955.661688] 7f83e378 3b000000 48658ac9 60000000 e89f00d0 3b400000 7fbd2040 419e0070 
[32955.663050] 60000000 60420000 e9240008 e9440000 <f92a0008> f9490000 f8840000 f8840008 
[32955.664260] ---[ end trace b20dd6fbb5b21935 ]---
[32955.677577]

== Comment: #1 - Kalpana Shetty <kalshett@xxxxxxxxxx> - 2014-08-05 23:54:12 ==
Setup details:
- MongoDB server on one VM (version: 2.6.2)
- YCSB workload running on one VM (YCSB version - ycsb-0.1.4)


[root@powerkvm5-lp1 ~]# uname -a
Linux powerkvm5-lp1.austin.ibm.com 3.10.42-2004.pkvm2_1_1.8.ppc64 #1 SMP Fri Jul 18 11:20:03 CDT 2014 ppc64 ppc64 ppc64 GNU/Linux

Guest OS: Ubuntu 14.10 
[root@powerkvm5-lp1 ~]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 3     kal_u10_ycsb                   running
 5     kal_u10_mongosrv               running

uname on Guest OS:
root@u10vm15:~# uname -a
Linux u10vm15 3.16.0-6-generic #11-Ubuntu SMP Mon Jul 28 02:00:45 UTC 2014 ppc64le ppc64le ppc64le GNU/Linux

root@u10vm15:~# lscpu
Architecture:          ppc64le
Byte Order:            Little Endian
CPU(s):                16
On-line CPU(s) list:   0-15
Thread(s) per core:    8
Core(s) per socket:    2
Socket(s):             1
NUMA node(s):          1
Model:                 IBM pSeries (emulated by qemu)
L1d cache:             64K
L1i cache:             32K
NUMA node0 CPU(s):     0-15

** Affects: linux (Ubuntu)
     Importance: Undecided
         Status: New


** Tags: architecture-ppc64le bugnameltc-114274 severity-critical targetmilestone-inin---
-- 
Running YCSB workload on MongoDB on Ubuntu 14.10 VM resulted in kernel bug
https://bugs.launchpad.net/bugs/1354024
You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.