kernel-packages team mailing list archive

Thread
Date
[Bug 1368245] [NEW] Random crash

To: kernel-packages@xxxxxxxxxxxxxxxxxxx
From: Leonid Samokhine <samokhine@xxxxxxxxxxx>
Date: Thu, 11 Sep 2014 13:56:06 -0000
Reply-to: Bug 1368245 <1368245@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

We are experiencing a random crashes (about ones in 2 days) after upgrading to Ubuntu 14.04.1 LTS.
The server starts killing processes due to lack of memory. If you wait long enough it hangs and only hard reboot can fix it. 

Here is I believe the first record in syslog when it starts:

Sep 11 11:21:08 pricing kernel: [158208.874463] ------------[ cut here ]------------
Sep 11 11:21:08 pricing kernel: [158208.874473] WARNING: CPU: 1 PID: 24257 at /build/buildd/linux-3.13.0/fs/dcache.c:362 d_shrink_del+0x71/0x80()
Sep 11 11:21:08 pricing kernel: [158208.874475] Modules linked in: xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG xt_limit xt_addrtype xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 x
t_conntrack ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack iptable_filter ip_tables x_tables isofs crct10dif_pclmul crc32_
pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd xfs libcrc32c raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid10 raid6_pq ra
id1 raid0 multipath linear
Sep 11 11:21:08 pricing kernel: [158208.874515] CPU: 1 PID: 24257 Comm: sh Not tainted 3.13.0-35-generic #62-Ubuntu
Sep 11 11:21:08 pricing kernel: [158208.874518]  0000000000000009 ffff880011055c78 ffffffff8171e320 0000000000000000
Sep 11 11:21:08 pricing kernel: [158208.874522]  ffff880011055cb0 ffffffff8106775d ffff88000b6a0900 ffff880011055d30
Sep 11 11:21:08 pricing kernel: [158208.874525]  ffff88000b6a0958 ffff88000b6a0900 ffff88000b6a0900 ffff880011055cc0
Sep 11 11:21:08 pricing kernel: [158208.874528] Call Trace:
Sep 11 11:21:08 pricing kernel: [158208.874534]  [<ffffffff8171e320>] dump_stack+0x45/0x56
Sep 11 11:21:08 pricing kernel: [158208.874539]  [<ffffffff8106775d>] warn_slowpath_common+0x7d/0xa0
Sep 11 11:21:08 pricing kernel: [158208.874542]  [<ffffffff8106783a>] warn_slowpath_null+0x1a/0x20
Sep 11 11:21:08 pricing kernel: [158208.874545]  [<ffffffff811d2af1>] d_shrink_del+0x71/0x80
Sep 11 11:21:08 pricing kernel: [158208.874548]  [<ffffffff811d3f64>] shrink_dentry_list+0x64/0xe0
Sep 11 11:21:08 pricing kernel: [158208.874551]  [<ffffffff811d4d78>] shrink_dcache_parent+0x28/0x70
Sep 11 11:21:08 pricing kernel: [158208.874556]  [<ffffffff81228686>] proc_flush_task+0xb6/0x1b0
Sep 11 11:21:08 pricing kernel: [158208.874560]  [<ffffffff810685e0>] release_task+0x30/0x460
Sep 11 11:21:08 pricing kernel: [158208.874564]  [<ffffffff8109e2f1>] ? thread_group_cputime_adjusted+0x41/0x50
Sep 11 11:21:08 pricing kernel: [158208.874567]  [<ffffffff810692ef>] wait_consider_task+0x8df/0xb20
Sep 11 11:21:08 pricing kernel: [158208.874570]  [<ffffffff81069630>] do_wait+0x100/0x240
Sep 11 11:21:08 pricing kernel: [158208.874573]  [<ffffffff8106a764>] SyS_wait4+0x64/0xe0
Sep 11 11:21:08 pricing kernel: [158208.874577]  [<ffffffff81068340>] ? task_stopped_code+0x60/0x60
Sep 11 11:21:08 pricing kernel: [158208.874580]  [<ffffffff8172ed6d>] system_call_fastpath+0x1a/0x1f
Sep 11 11:21:08 pricing kernel: [158208.874582] ---[ end trace 5777f83b2241c020 ]---
Sep 11 11:21:35 pricing kernel: [158236.096055] BUG: soft lockup - CPU#1 stuck for 22s! [sh:24257]
Sep 11 11:21:35 pricing kernel: [158236.096055] Modules linked in: xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG xt_limit xt_addrtype xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 x
t_conntrack ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack iptable_filter ip_tables x_tables isofs crct10dif_pclmul crc32_
pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd xfs libcrc32c raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid10 raid6_pq ra
id1 raid0 multipath linear
Sep 11 11:21:35 pricing kernel: [158236.096055] CPU: 1 PID: 24257 Comm: sh Tainted: G        W    3.13.0-35-generic #62-Ubuntu
Sep 11 11:21:35 pricing kernel: [158236.096055] task: ffff8800de5f8000 ti: ffff880011054000 task.ti: ffff880011054000
Sep 11 11:21:35 pricing kernel: [158236.096055] RIP: e030:[<ffffffff811d2a80>]  [<ffffffff811d2a80>] d_shrink_del+0x0/0x80
Sep 11 11:21:35 pricing kernel: [158236.096055] RSP: e02b:ffff880011055ce0  EFLAGS: 00000246
Sep 11 11:21:35 pricing kernel: [158236.096055] RAX: ffff88000b6a0980 RBX: ffff88000b6a0980 RCX: ffff88000b6a0980
Sep 11 11:21:35 pricing kernel: [158236.096055] RDX: 000000000000fbe4 RSI: 00000000fbe4fbe2 RDI: ffff88000b6a0900
Sep 11 11:21:35 pricing kernel: [158236.096055] RBP: ffff880011055d18 R08: 0000000000000000 R09: 00000000000001c5
Sep 11 11:21:35 pricing kernel: [158236.096055] R10: 0000000000000000 R11: ffff8800110559a6 R12: ffff880011055d30
Sep 11 11:21:35 pricing kernel: [158236.096055] R13: ffff88000b6a0958 R14: ffff88000b6a0900 R15: ffff88000b6a0900
Sep 11 11:21:35 pricing kernel: [158236.096055] FS:  00007f1611926740(0000) GS:ffff8803bdc80000(0000) knlGS:0000000000000000
Sep 11 11:21:35 pricing kernel: [158236.096055] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Sep 11 11:21:35 pricing kernel: [158236.096055] CR2: 00007f079beb1d50 CR3: 000000000cfcf000 CR4: 0000000000002660
Sep 11 11:21:35 pricing kernel: [158236.096055] Stack:
Sep 11 11:21:35 pricing kernel: [158236.096055]  ffffffff811d3f64 ffff88000b6a0980 ffff880011055d30 ffff88010764f3c0
Sep 11 11:21:35 pricing kernel: [158236.096055]  ffff8803a6d603e0 ffff88010764f3c0 0000000000005ec2 ffff880011055d58
Sep 11 11:21:35 pricing kernel: [158236.096055]  ffffffff811d4d78 ffff88010764f3c0 ffff88000b6a0b00 ffff88000b6a0980
Sep 11 11:21:35 pricing kernel: [158236.096055] Call Trace:
Sep 11 11:21:35 pricing kernel: [158236.096055]  [<ffffffff811d3f64>] ? shrink_dentry_list+0x64/0xe0
Sep 11 11:21:35 pricing kernel: [158236.096055]  [<ffffffff811d4d78>] shrink_dcache_parent+0x28/0x70
Sep 11 11:21:35 pricing kernel: [158236.096055]  [<ffffffff81228686>] proc_flush_task+0xb6/0x1b0

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: linux-image-3.13.0-35-generic 3.13.0-35.62
ProcVersionSignature: Ubuntu 3.13.0-35.62-generic 3.13.11.6
Uname: Linux 3.13.0-35-generic x86_64
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116,  1 Sep 11 12:29 seq
 crw-rw---- 1 root audio 116, 33 Sep 11 12:29 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.14.1-0ubuntu3.3
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
CurrentDmesg: [   45.163030] init: plymouth-upstart-bridge main process ended, respawning
Date: Thu Sep 11 13:41:04 2014
Ec2AMI: ami-3fec7956
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: us-east-1c
Ec2InstanceType: m3.xlarge
Ec2Kernel: aki-88aa75e1
Ec2Ramdisk: unavailable
IwConfig:
 eth0      no wireless extensions.
 
 lo        no wireless extensions.
Lspci:
 
Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
PciMultimedia:
 
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB:
 
ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0
RelatedPackageVersions:
 linux-restricted-modules-3.13.0-35-generic N/A
 linux-backports-modules-3.13.0-35-generic  N/A
 linux-firmware                             1.127.5
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: Upgraded to trusty on 2014-09-07 (4 days ago)

** Affects: linux (Ubuntu)
     Importance: Undecided
         Status: Incomplete


** Tags: amd64 apport-bug ec2-images trusty

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1368245

Title:
  Random crash

Status in “linux” package in Ubuntu:
  Incomplete

Bug description:
  We are experiencing a random crashes (about ones in 2 days) after upgrading to Ubuntu 14.04.1 LTS.
  The server starts killing processes due to lack of memory. If you wait long enough it hangs and only hard reboot can fix it. 

  Here is I believe the first record in syslog when it starts:

  Sep 11 11:21:08 pricing kernel: [158208.874463] ------------[ cut here ]------------
  Sep 11 11:21:08 pricing kernel: [158208.874473] WARNING: CPU: 1 PID: 24257 at /build/buildd/linux-3.13.0/fs/dcache.c:362 d_shrink_del+0x71/0x80()
  Sep 11 11:21:08 pricing kernel: [158208.874475] Modules linked in: xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG xt_limit xt_addrtype xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 x
  t_conntrack ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack iptable_filter ip_tables x_tables isofs crct10dif_pclmul crc32_
  pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd xfs libcrc32c raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid10 raid6_pq ra
  id1 raid0 multipath linear
  Sep 11 11:21:08 pricing kernel: [158208.874515] CPU: 1 PID: 24257 Comm: sh Not tainted 3.13.0-35-generic #62-Ubuntu
  Sep 11 11:21:08 pricing kernel: [158208.874518]  0000000000000009 ffff880011055c78 ffffffff8171e320 0000000000000000
  Sep 11 11:21:08 pricing kernel: [158208.874522]  ffff880011055cb0 ffffffff8106775d ffff88000b6a0900 ffff880011055d30
  Sep 11 11:21:08 pricing kernel: [158208.874525]  ffff88000b6a0958 ffff88000b6a0900 ffff88000b6a0900 ffff880011055cc0
  Sep 11 11:21:08 pricing kernel: [158208.874528] Call Trace:
  Sep 11 11:21:08 pricing kernel: [158208.874534]  [<ffffffff8171e320>] dump_stack+0x45/0x56
  Sep 11 11:21:08 pricing kernel: [158208.874539]  [<ffffffff8106775d>] warn_slowpath_common+0x7d/0xa0
  Sep 11 11:21:08 pricing kernel: [158208.874542]  [<ffffffff8106783a>] warn_slowpath_null+0x1a/0x20
  Sep 11 11:21:08 pricing kernel: [158208.874545]  [<ffffffff811d2af1>] d_shrink_del+0x71/0x80
  Sep 11 11:21:08 pricing kernel: [158208.874548]  [<ffffffff811d3f64>] shrink_dentry_list+0x64/0xe0
  Sep 11 11:21:08 pricing kernel: [158208.874551]  [<ffffffff811d4d78>] shrink_dcache_parent+0x28/0x70
  Sep 11 11:21:08 pricing kernel: [158208.874556]  [<ffffffff81228686>] proc_flush_task+0xb6/0x1b0
  Sep 11 11:21:08 pricing kernel: [158208.874560]  [<ffffffff810685e0>] release_task+0x30/0x460
  Sep 11 11:21:08 pricing kernel: [158208.874564]  [<ffffffff8109e2f1>] ? thread_group_cputime_adjusted+0x41/0x50
  Sep 11 11:21:08 pricing kernel: [158208.874567]  [<ffffffff810692ef>] wait_consider_task+0x8df/0xb20
  Sep 11 11:21:08 pricing kernel: [158208.874570]  [<ffffffff81069630>] do_wait+0x100/0x240
  Sep 11 11:21:08 pricing kernel: [158208.874573]  [<ffffffff8106a764>] SyS_wait4+0x64/0xe0
  Sep 11 11:21:08 pricing kernel: [158208.874577]  [<ffffffff81068340>] ? task_stopped_code+0x60/0x60
  Sep 11 11:21:08 pricing kernel: [158208.874580]  [<ffffffff8172ed6d>] system_call_fastpath+0x1a/0x1f
  Sep 11 11:21:08 pricing kernel: [158208.874582] ---[ end trace 5777f83b2241c020 ]---
  Sep 11 11:21:35 pricing kernel: [158236.096055] BUG: soft lockup - CPU#1 stuck for 22s! [sh:24257]
  Sep 11 11:21:35 pricing kernel: [158236.096055] Modules linked in: xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG xt_limit xt_addrtype xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 x
  t_conntrack ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack iptable_filter ip_tables x_tables isofs crct10dif_pclmul crc32_
  pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd xfs libcrc32c raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid10 raid6_pq ra
  id1 raid0 multipath linear
  Sep 11 11:21:35 pricing kernel: [158236.096055] CPU: 1 PID: 24257 Comm: sh Tainted: G        W    3.13.0-35-generic #62-Ubuntu
  Sep 11 11:21:35 pricing kernel: [158236.096055] task: ffff8800de5f8000 ti: ffff880011054000 task.ti: ffff880011054000
  Sep 11 11:21:35 pricing kernel: [158236.096055] RIP: e030:[<ffffffff811d2a80>]  [<ffffffff811d2a80>] d_shrink_del+0x0/0x80
  Sep 11 11:21:35 pricing kernel: [158236.096055] RSP: e02b:ffff880011055ce0  EFLAGS: 00000246
  Sep 11 11:21:35 pricing kernel: [158236.096055] RAX: ffff88000b6a0980 RBX: ffff88000b6a0980 RCX: ffff88000b6a0980
  Sep 11 11:21:35 pricing kernel: [158236.096055] RDX: 000000000000fbe4 RSI: 00000000fbe4fbe2 RDI: ffff88000b6a0900
  Sep 11 11:21:35 pricing kernel: [158236.096055] RBP: ffff880011055d18 R08: 0000000000000000 R09: 00000000000001c5
  Sep 11 11:21:35 pricing kernel: [158236.096055] R10: 0000000000000000 R11: ffff8800110559a6 R12: ffff880011055d30
  Sep 11 11:21:35 pricing kernel: [158236.096055] R13: ffff88000b6a0958 R14: ffff88000b6a0900 R15: ffff88000b6a0900
  Sep 11 11:21:35 pricing kernel: [158236.096055] FS:  00007f1611926740(0000) GS:ffff8803bdc80000(0000) knlGS:0000000000000000
  Sep 11 11:21:35 pricing kernel: [158236.096055] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
  Sep 11 11:21:35 pricing kernel: [158236.096055] CR2: 00007f079beb1d50 CR3: 000000000cfcf000 CR4: 0000000000002660
  Sep 11 11:21:35 pricing kernel: [158236.096055] Stack:
  Sep 11 11:21:35 pricing kernel: [158236.096055]  ffffffff811d3f64 ffff88000b6a0980 ffff880011055d30 ffff88010764f3c0
  Sep 11 11:21:35 pricing kernel: [158236.096055]  ffff8803a6d603e0 ffff88010764f3c0 0000000000005ec2 ffff880011055d58
  Sep 11 11:21:35 pricing kernel: [158236.096055]  ffffffff811d4d78 ffff88010764f3c0 ffff88000b6a0b00 ffff88000b6a0980
  Sep 11 11:21:35 pricing kernel: [158236.096055] Call Trace:
  Sep 11 11:21:35 pricing kernel: [158236.096055]  [<ffffffff811d3f64>] ? shrink_dentry_list+0x64/0xe0
  Sep 11 11:21:35 pricing kernel: [158236.096055]  [<ffffffff811d4d78>] shrink_dcache_parent+0x28/0x70
  Sep 11 11:21:35 pricing kernel: [158236.096055]  [<ffffffff81228686>] proc_flush_task+0xb6/0x1b0

  ProblemType: Bug
  DistroRelease: Ubuntu 14.04
  Package: linux-image-3.13.0-35-generic 3.13.0-35.62
  ProcVersionSignature: Ubuntu 3.13.0-35.62-generic 3.13.11.6
  Uname: Linux 3.13.0-35-generic x86_64
  AlsaDevices:
   total 0
   crw-rw---- 1 root audio 116,  1 Sep 11 12:29 seq
   crw-rw---- 1 root audio 116, 33 Sep 11 12:29 timer
  AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
  ApportVersion: 2.14.1-0ubuntu3.3
  Architecture: amd64
  ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
  AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
  CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
  CurrentDmesg: [   45.163030] init: plymouth-upstart-bridge main process ended, respawning
  Date: Thu Sep 11 13:41:04 2014
  Ec2AMI: ami-3fec7956
  Ec2AMIManifest: (unknown)
  Ec2AvailabilityZone: us-east-1c
  Ec2InstanceType: m3.xlarge
  Ec2Kernel: aki-88aa75e1
  Ec2Ramdisk: unavailable
  IwConfig:
   eth0      no wireless extensions.
   
   lo        no wireless extensions.
  Lspci:
   
  Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
  PciMultimedia:
   
  ProcEnviron:
   TERM=xterm
   PATH=(custom, no user)
   XDG_RUNTIME_DIR=<set>
   LANG=en_US.UTF-8
   SHELL=/bin/bash
  ProcFB:
   
  ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0
  RelatedPackageVersions:
   linux-restricted-modules-3.13.0-35-generic N/A
   linux-backports-modules-3.13.0-35-generic  N/A
   linux-firmware                             1.127.5
  RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
  SourcePackage: linux
  UpgradeStatus: Upgraded to trusty on 2014-09-07 (4 days ago)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1368245/+subscriptions
Follow ups

[Bug 1368245] Re: Random crash
From: Dave Chiluk, 2014-09-11
[Bug 1368245] Missing required logs.
From: Brad Figg, 2014-09-11
[Bug 1368245] [NEW] Random crash
From: Leonid Samokhine, 2014-09-11
References

[Bug 1368245] [NEW] Random crash
From: Leonid Samokhine, 2014-09-11