kernel-packages team mailing list archive

Thread
Date
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam

To: kernel-packages@xxxxxxxxxxxxxxxxxxx
From: Rafael David Tinoco <rafael.tinoco@xxxxxxxxxxxxx>
Date: Sun, 22 Nov 2015 22:40:59 -0000
Reply-to: Bug 1505564 <1505564@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Description changed:

- Hi,
+ Some of our nova compute hosts regularly freeze, sometimes for a few
+ hours, with kern.log getting spammed with:
  
- Some of our nova compute hosts regularly freeze, sometimes for a few hours, with kern.log getting spammed with :
  block nbdX: Attempted send on closed socket
  
- and a few "CPU soft lockup" messages (see attached log). This clears up when the queue gets cleared, eg :
+ and a few "CPU soft lockup" messages (see attached log). This clears up
+ when the queue gets cleared, eg :
+ 
  block nbdX: queue cleared
  
  trusty hosts with kernel version 3.19.0-30-generic.
- 
- Note that timestamps from kern.log appears to be wrong, it looks like
- the messages are being held, and then all delivered at once when the
- kernel unfreezes.
- 
- Attaching apport files from 2 hosts below.
- --- 
- AlsaDevices:
-  total 0
-  crw-rw---- 1 root audio 116,  1 Oct  9 13:14 seq
-  crw-rw---- 1 root audio 116, 33 Oct  9 13:14 timer
- AplayDevices: Error: [Errno 2] No such file or directory
- ApportVersion: 2.14.1-0ubuntu3.15
- Architecture: amd64
- ArecordDevices: Error: [Errno 2] No such file or directory
- AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
- DistroRelease: Ubuntu 14.04
- IwConfig: Error: [Errno 2] No such file or directory
- MachineType: HP ProLiant DL385 G7
- Package: linux (not installed)
- PciMultimedia:
-  
- ProcEnviron:
-  TERM=screen-256color
-  PATH=(custom, no user)
-  LANG=en_US.UTF-8
-  SHELL=/bin/bash
- ProcFB: 0 radeondrmfb
- ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.19.0-30-generic root=UUID=1bd039df-c419-4cb7-b1ad-fe004d55ccd4 ro console=tty0 console=ttyS1,38400 nosplash
- ProcVersionSignature: Ubuntu 3.19.0-30.34~14.04.1-generic 3.19.8-ckt6
- RelatedPackageVersions:
-  linux-restricted-modules-3.19.0-30-generic N/A
-  linux-backports-modules-3.19.0-30-generic  N/A
-  linux-firmware                             1.127.15
- RfKill: Error: [Errno 2] No such file or directory
- Tags:  trusty uec-images
- Uname: Linux 3.19.0-30-generic x86_64
- UpgradeStatus: No upgrade log present (probably fresh install)
- UserGroups:
-  
- _MarkForUpload: True
- dmi.bios.date: 12/08/2012
- dmi.bios.vendor: HP
- dmi.bios.version: A18
- dmi.chassis.type: 23
- dmi.chassis.vendor: HP
- dmi.modalias: dmi:bvnHP:bvrA18:bd12/08/2012:svnHP:pnProLiantDL385G7:pvr:cvnHP:ct23:cvr:
- dmi.product.name: ProLiant DL385 G7
- dmi.sys.vendor: HP
- --- 
- AlsaDevices:
-  total 0
-  crw-rw---- 1 root audio 116,  1 Oct  9 12:37 seq
-  crw-rw---- 1 root audio 116, 33 Oct  9 12:37 timer
- AplayDevices: Error: [Errno 2] No such file or directory
- ApportVersion: 2.14.1-0ubuntu3.15
- Architecture: amd64
- ArecordDevices: Error: [Errno 2] No such file or directory
- AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
- DistroRelease: Ubuntu 14.04
- IwConfig: Error: [Errno 2] No such file or directory
- MachineType: HP ProLiant DL360p Gen8
- Package: linux (not installed)
- PciMultimedia:
-  
- ProcEnviron:
-  TERM=screen-256color
-  PATH=(custom, no user)
-  LANG=en_US.UTF-8
-  SHELL=/bin/bash
- ProcFB: 0 VESA VGA
- ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.19.0-30-generic root=UUID=f3c6cae8-09dc-4607-8675-c9123ea9c9fd ro console=tty0 console=ttyS1,38400 nosplash
- ProcVersionSignature: Ubuntu 3.19.0-30.34~14.04.1-generic 3.19.8-ckt6
- RelatedPackageVersions:
-  linux-restricted-modules-3.19.0-30-generic N/A
-  linux-backports-modules-3.19.0-30-generic  N/A
-  linux-firmware                             1.127.15
- RfKill: Error: [Errno 2] No such file or directory
- StagingDrivers: visorchannel visorutil
- Tags:  trusty uec-images staging
- Uname: Linux 3.19.0-30-generic x86_64
- UpgradeStatus: No upgrade log present (probably fresh install)
- UserGroups:
-  
- _MarkForUpload: True
- dmi.bios.date: 11/14/2013
- dmi.bios.vendor: HP
- dmi.bios.version: P71
- dmi.chassis.type: 23
- dmi.chassis.vendor: HP
- dmi.modalias: dmi:bvnHP:bvrP71:bd11/14/2013:svnHP:pnProLiantDL360pGen8:pvr:cvnHP:ct23:cvr:
- dmi.product.name: ProLiant DL360p Gen8
- dmi.sys.vendor: HP
- --- 
- AlsaDevices:
-  total 0
-  crw-rw---- 1 root audio 116,  1 Oct 28 13:07 seq
-  crw-rw---- 1 root audio 116, 33 Oct 28 13:07 timer
- AplayDevices: Error: [Errno 2] No such file or directory
- ApportVersion: 2.14.1-0ubuntu3.16
- Architecture: amd64
- ArecordDevices: Error: [Errno 2] No such file or directory
- AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
- DistroRelease: Ubuntu 14.04
- IwConfig: Error: [Errno 2] No such file or directory
- MachineType: HP ProLiant DL360p Gen8
- Package: linux (not installed)
- PciMultimedia:
-  
- ProcEnviron:
-  TERM=screen-256color
-  PATH=(custom, no user)
-  LANG=en_US.UTF-8
-  SHELL=/bin/bash
- ProcFB: 0 VESA VGA
- ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-29-generic root=UUID=46ac2c1e-5f16-45bd-b383-e952f78fd142 ro console=tty0 console=ttyS1,38400 nosplash crashkernel=384M-:512M
- ProcVersionSignature: Ubuntu 3.13.0-29.53-generic 3.13.11.2
- RelatedPackageVersions:
-  linux-restricted-modules-3.13.0-29-generic N/A
-  linux-backports-modules-3.13.0-29-generic  N/A
-  linux-firmware                             1.127.15
- RfKill: Error: [Errno 2] No such file or directory
- Tags:  trusty uec-images
- Uname: Linux 3.13.0-29-generic x86_64
- UpgradeStatus: No upgrade log present (probably fresh install)
- UserGroups:
-  
- _MarkForUpload: True
- dmi.bios.date: 11/14/2013
- dmi.bios.vendor: HP
- dmi.bios.version: P71
- dmi.chassis.type: 23
- dmi.chassis.vendor: HP
- dmi.modalias: dmi:bvnHP:bvrP71:bd11/14/2013:svnHP:pnProLiantDL360pGen8:pvr:cvnHP:ct23:cvr:
- dmi.product.name: ProLiant DL360p Gen8
- dmi.sys.vendor: HP
- --- 
- AlsaDevices:
-  total 0
-  crw-rw---- 1 root audio 116,  1 Oct 28 16:31 seq
-  crw-rw---- 1 root audio 116, 33 Oct 28 16:31 timer
- AplayDevices: Error: [Errno 2] No such file or directory
- ApportVersion: 2.14.1-0ubuntu3.18
- Architecture: amd64
- ArecordDevices: Error: [Errno 2] No such file or directory
- AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
- DistroRelease: Ubuntu 14.04
- IwConfig: Error: [Errno 2] No such file or directory
- MachineType: HP ProLiant DL385 G7
- Package: linux (not installed)
- PciMultimedia:
-  
- ProcEnviron:
-  TERM=screen-256color
-  PATH=(custom, no user)
-  LANG=en_US.UTF-8
-  SHELL=/bin/bash
- ProcFB: 0 radeondrmfb
- ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-24-generic root=UUID=29988d72-90fe-4329-a0c1-0e3bfb88beab ro console=tty0 console=ttyS1,38400 nosplash crashkernel=384M-:512M
- ProcVersionSignature: Ubuntu 3.13.0-24.47-generic 3.13.9
- RelatedPackageVersions:
-  linux-restricted-modules-3.13.0-24-generic N/A
-  linux-backports-modules-3.13.0-24-generic  N/A
-  linux-firmware                             1.127.15
- RfKill: Error: [Errno 2] No such file or directory
- Tags:  trusty uec-images
- Uname: Linux 3.13.0-24-generic x86_64
- UpgradeStatus: No upgrade log present (probably fresh install)
- UserGroups:
-  
- WifiSyslog:
-  Oct 29 07:58:52 druk kernel: [55713.346292] nr_pdflush_threads exported in /proc is scheduled for removal
-  Oct 29 07:58:52 druk kernel: [55713.346523] sysctl: The scan_unevictable_pages sysctl/node-interface has been disabled for lack of a legitimate use case.  If you have one, please send an email to linux-mm@xxxxxxxxx.
- _MarkForUpload: True
- dmi.bios.date: 02/02/2014
- dmi.bios.vendor: HP
- dmi.bios.version: A18
- dmi.chassis.type: 23
- dmi.chassis.vendor: HP
- dmi.modalias: dmi:bvnHP:bvrA18:bd02/02/2014:svnHP:pnProLiantDL385G7:pvr:cvnHP:ct23:cvr:
- dmi.product.name: ProLiant DL385 G7
- dmi.sys.vendor: HP
- --- 
- AlsaDevices:
-  total 0
-  crw-rw---- 1 root audio 116,  1 Oct 28 19:56 seq
-  crw-rw---- 1 root audio 116, 33 Oct 28 19:56 timer
- AplayDevices: Error: [Errno 2] No such file or directory
- ApportVersion: 2.14.1-0ubuntu3.18
- Architecture: amd64
- ArecordDevices: Error: [Errno 2] No such file or directory
- AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
- DistroRelease: Ubuntu 14.04
- IwConfig: Error: [Errno 2] No such file or directory
- MachineType: HP ProLiant DL360p Gen8
- Package: linux (not installed)
- PciMultimedia:
-  
- ProcEnviron:
-  TERM=screen-256color
-  PATH=(custom, no user)
-  LANG=en_US.UTF-8
-  SHELL=/bin/bash
- ProcFB: 0 VESA VGA
- ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-29-generic root=UUID=f3c6cae8-09dc-4607-8675-c9123ea9c9fd ro console=tty0 console=ttyS1,38400 nosplash crashkernel=384M-:512M
- ProcVersionSignature: Ubuntu 3.13.0-29.53-generic 3.13.11.2
- RelatedPackageVersions:
-  linux-restricted-modules-3.13.0-29-generic N/A
-  linux-backports-modules-3.13.0-29-generic  N/A
-  linux-firmware                             1.127.15
- RfKill: Error: [Errno 2] No such file or directory
- Tags:  trusty uec-images
- Uname: Linux 3.13.0-29-generic x86_64
- UpgradeStatus: No upgrade log present (probably fresh install)
- UserGroups:
-  
- WifiSyslog:
-  Oct 29 07:41:34 orlo kernel: [42337.380135] qbrce3298b1-cb: port 2(tapce3298b1-cb) entered disabled state
-  Oct 29 07:41:34 orlo kernel: [42337.380543] device tapce3298b1-cb left promiscuous mode
-  Oct 29 07:41:34 orlo kernel: [42337.380580] qbrce3298b1-cb: port 2(tapce3298b1-cb) entered disabled state
-  Oct 29 07:41:35 orlo kernel: [42338.223036] type=1400 audit(1446104495.424:137): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="libvirt-80a3c754-9f62-4011-baea-b3a8f37d3746" pid=11554 comm="apparmor_parser"
-  Oct 29 07:41:35 orlo kernel: [42338.308192] qbrce3298b1-cb: port 1(qvbce3298b1-cb) entered disabled state
- _MarkForUpload: True
- dmi.bios.date: 11/14/2013
- dmi.bios.vendor: HP
- dmi.bios.version: P71
- dmi.chassis.type: 23
- dmi.chassis.vendor: HP
- dmi.modalias: dmi:bvnHP:bvrP71:bd11/14/2013:svnHP:pnProLiantDL360pGen8:pvr:cvnHP:ct23:cvr:
- dmi.product.name: ProLiant DL360p Gen8
- dmi.sys.vendor: HP
- --- 
- AlsaDevices:
-  total 0
-  crw-rw---- 1 root audio 116,  1 Oct 29 21:10 seq
-  crw-rw---- 1 root audio 116, 33 Oct 29 21:10 timer
- AplayDevices: Error: [Errno 2] No such file or directory
- ApportVersion: 2.14.1-0ubuntu3.18
- Architecture: amd64
- ArecordDevices: Error: [Errno 2] No such file or directory
- AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
- DistroRelease: Ubuntu 14.04
- IwConfig: Error: [Errno 2] No such file or directory
- MachineType: HP ProLiant DL360p Gen8
- Package: linux (not installed)
- PciMultimedia:
-  
- ProcEnviron:
-  TERM=screen-256color
-  PATH=(custom, no user)
-  LANG=en_US.UTF-8
-  SHELL=/bin/bash
- ProcFB: 0 VESA VGA
- ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.19.0-31-generic root=UUID=f3c6cae8-09dc-4607-8675-c9123ea9c9fd ro console=tty0 console=ttyS1,38400 nosplash crashkernel=384M-:512M
- ProcVersionSignature: Ubuntu 3.19.0-31.36~14.04.1-generic 3.19.8-ckt7
- RelatedPackageVersions:
-  linux-restricted-modules-3.19.0-31-generic N/A
-  linux-backports-modules-3.19.0-31-generic  N/A
-  linux-firmware                             1.127.16
- RfKill: Error: [Errno 2] No such file or directory
- StagingDrivers: visorchannel visorutil
- Tags:  trusty uec-images staging
- Uname: Linux 3.19.0-31-generic x86_64
- UpgradeStatus: No upgrade log present (probably fresh install)
- UserGroups:
-  
- _MarkForUpload: True
- dmi.bios.date: 11/14/2013
- dmi.bios.vendor: HP
- dmi.bios.version: P71
- dmi.chassis.type: 23
- dmi.chassis.vendor: HP
- dmi.modalias: dmi:bvnHP:bvrP71:bd11/14/2013:svnHP:pnProLiantDL360pGen8:pvr:cvnHP:ct23:cvr:
- dmi.product.name: ProLiant DL360p Gen8
- dmi.sys.vendor: HP
- --- 
- AlsaDevices:
-  total 0
-  crw-rw---- 1 root audio 116,  1 Oct 30 12:31 seq
-  crw-rw---- 1 root audio 116, 33 Oct 30 12:31 timer
- AplayDevices: Error: [Errno 2] No such file or directory
- ApportVersion: 2.14.1-0ubuntu3.18
- Architecture: amd64
- ArecordDevices: Error: [Errno 2] No such file or directory
- AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
- DistroRelease: Ubuntu 14.04
- IwConfig: Error: [Errno 2] No such file or directory
- MachineType: HP ProLiant DL360p Gen8
- Package: linux (not installed)
- PciMultimedia:
-  
- ProcEnviron:
-  TERM=screen-256color
-  PATH=(custom, no user)
-  LANG=en_US.UTF-8
-  SHELL=/bin/bash
- ProcFB: 0 VESA VGA
- ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.19.0-31-generic root=UUID=46ac2c1e-5f16-45bd-b383-e952f78fd142 ro console=tty0 console=ttyS1,38400 nosplash crashkernel=384M-:512M
- ProcVersionSignature: Ubuntu 3.19.0-31.36~14.04.1-generic 3.19.8-ckt7
- RelatedPackageVersions:
-  linux-restricted-modules-3.19.0-31-generic N/A
-  linux-backports-modules-3.19.0-31-generic  N/A
-  linux-firmware                             1.127.16
- RfKill: Error: [Errno 2] No such file or directory
- StagingDrivers: visorchannel visorutil
- Tags:  trusty uec-images staging
- Uname: Linux 3.19.0-31-generic x86_64
- UpgradeStatus: No upgrade log present (probably fresh install)
- UserGroups:
-  
- _MarkForUpload: True
- dmi.bios.date: 11/14/2013
- dmi.bios.vendor: HP
- dmi.bios.version: P71
- dmi.chassis.type: 23
- dmi.chassis.vendor: HP
- dmi.modalias: dmi:bvnHP:bvrP71:bd11/14/2013:svnHP:pnProLiantDL360pGen8:pvr:cvnHP:ct23:cvr:
- dmi.product.name: ProLiant DL360p Gen8
- dmi.sys.vendor: HP
- --- 
- AlsaDevices:
-  total 0
-  crw-rw---- 1 root audio 116,  1 Nov 17 12:47 seq
-  crw-rw---- 1 root audio 116, 33 Nov 17 12:47 timer
- AplayDevices: Error: [Errno 2] No such file or directory
- ApportVersion: 2.14.1-0ubuntu3.19
- Architecture: amd64
- ArecordDevices: Error: [Errno 2] No such file or directory
- AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
- DistroRelease: Ubuntu 14.04
- IwConfig: Error: [Errno 2] No such file or directory
- MachineType: HP ProLiant DL385 G7
- Package: linux (not installed)
- PciMultimedia:
-  
- ProcEnviron:
-  TERM=screen-256color
-  PATH=(custom, no user)
-  LANG=en_US.UTF-8
-  SHELL=/bin/bash
- ProcFB: 0 radeondrmfb
- ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.19.0-33-generic root=UUID=29988d72-90fe-4329-a0c1-0e3bfb88beab ro console=tty0 console=ttyS1,38400 nosplash crashkernel=384M-:512M nox2apic intremap=off
- ProcVersionSignature: Ubuntu 3.19.0-33.38~14.04.1-generic 3.19.8-ckt7
- RelatedPackageVersions:
-  linux-restricted-modules-3.19.0-33-generic N/A
-  linux-backports-modules-3.19.0-33-generic  N/A
-  linux-firmware                             1.127.18
- RfKill: Error: [Errno 2] No such file or directory
- Tags:  trusty uec-images
- Uname: Linux 3.19.0-33-generic x86_64
- UpgradeStatus: No upgrade log present (probably fresh install)
- UserGroups:
-  
- _MarkForUpload: True
- dmi.bios.date: 02/02/2014
- dmi.bios.vendor: HP
- dmi.bios.version: A18
- dmi.chassis.type: 23
- dmi.chassis.vendor: HP
- dmi.modalias: dmi:bvnHP:bvrA18:bd02/02/2014:svnHP:pnProLiantDL385G7:pvr:cvnHP:ct23:cvr:
- dmi.product.name: ProLiant DL385 G7
- dmi.sys.vendor: HP

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1505564

Title:
  Soft lockup with "block nbdX: Attempted send on closed socket" spam

Status in linux package in Ubuntu:
  In Progress

Bug description:
  Some of our nova compute hosts regularly freeze, sometimes for a few
  hours, with kern.log getting spammed with:

  block nbdX: Attempted send on closed socket

  and a few "CPU soft lockup" messages (see attached log). This clears
  up when the queue gets cleared, eg :

  block nbdX: queue cleared

  trusty hosts with kernel version 3.19.0-30-generic.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions
References

[Bug 1505564] [NEW] Soft lockup with "block nbdX: Attempted send on closed socket" spam
From: Junien Fridrick, 2015-10-13