kernel-packages team mailing list archive

[Nick Moffitt <nick.moffitt@xxxxxxxxxxxxx>]

Public bug reported:

We regularly use bcache to use SSDs as an extra block cache layer for
performance on large working sets (archive mirrors, large databases,
etc).  Until now we have had good success with bcache on precise LTS
with HWE kernels up to Saucy's.

Unfortunately when we mount a bcache-backed volume on a Trusty server,
it immediately loses contact with bcache_writebac:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                            
  501 root      20   0       0      0      0 D   0.0  0.0   0:00.00 bcache_writebac                    

kern.log shows it losing its mind pretty much immediately on startup:

    Aug 12 02:22:15 somehostname kernel: [ 6483.477618] md: bind<sdc>
    Aug 12 02:22:15 somehostname kernel: [ 6483.478599] md: bind<sdd>
    Aug 12 02:22:15 somehostname kernel: [ 6483.482016] md/raid0:md0: md_size is 781443072 sectors.
    Aug 12 02:22:15 somehostname kernel: [ 6483.482024] md: RAID0 configuration for md0 - 1 zone
    Aug 12 02:22:15 somehostname kernel: [ 6483.482026] md: zone0=[sdc/sdd]
    Aug 12 02:22:15 somehostname kernel: [ 6483.482033]       zone-offset=         0KB, device-offset=         0KB, size= 390721536KB
    Aug 12 02:22:15 somehostname kernel: [ 6483.482036] 
    Aug 12 02:22:15 somehostname kernel: [ 6483.482055] md0: detected capacity change from 0 to 400098852864
    Aug 12 02:22:15 somehostname kernel: [ 6483.487644]  md0: unknown partition table
    Aug 12 02:26:55 somehostname kernel: [ 6763.657249] bio: create slab <bio-2> at 2
    Aug 12 02:26:55 somehostname kernel: [ 6763.658018] bcache: run_cache_set() invalidating existing data
    Aug 12 02:26:55 somehostname kernel: [ 6763.695447] bcache: register_cache() registered cache device md0
    Aug 12 02:26:55 somehostname kernel: [ 6763.696262] bcache: register_bdev() registered backing device sdb
    Aug 12 02:26:55 somehostname kernel: [ 6763.697833] bcache: bch_cached_dev_attach() Caching sdb as bcache0 on set 13dcf71e-73ea-49f7-8cd6-fb8182c3cc17
    Aug 12 02:27:24 somehostname kernel: [ 6793.114287] EXT4-fs (bcache0): mounted filesystem with ordered data mode. Opts: (null)
    Aug 12 02:30:16 somehostname kernel: [ 6964.514425] INFO: task bcache_writebac:29721 blocked for more than 120 seconds.
    Aug 12 02:30:16 somehostname kernel: [ 6964.608861]       Not tainted 3.13.0-24-generic #47-Ubuntu
    Aug 12 02:30:16 somehostname kernel: [ 6964.679403] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
    Aug 12 02:30:16 somehostname kernel: [ 6964.779270] bcache_writebac D ffff8817dfb34440     0 29721      2 0x00000000
    Aug 12 02:30:16 somehostname kernel: [ 6964.779278]  ffff882fafe57eb8 0000000000000002 ffff882fb100afe0 ffff882fafe57fd8
    Aug 12 02:30:16 somehostname kernel: [ 6964.779286]  0000000000014440 0000000000014440 ffff882fb100afe0 ffff882fb7bcac80
    Aug 12 02:30:16 somehostname kernel: [ 6964.779293]  ffff882fb11c0000 ffffffffa03f5840 0000000000000000 0000000000000000
    Aug 12 02:30:16 somehostname kernel: [ 6964.779301] Call Trace:
    Aug 12 02:30:16 somehostname kernel: [ 6964.779335]  [<ffffffffa03f5840>] ? read_dirty+0x400/0x400 [bcache]
    Aug 12 02:30:16 somehostname kernel: [ 6964.779346]  [<ffffffff81719ee9>] schedule+0x29/0x70
    Aug 12 02:30:16 somehostname kernel: [ 6964.779356]  [<ffffffff8108b2ed>] kthread+0xad/0xf0
    Aug 12 02:30:16 somehostname kernel: [ 6964.779368]  [<ffffffff8108b240>] ? kthread_create_on_node+0x1d0/0x1d0
    Aug 12 02:30:16 somehostname kernel: [ 6964.779379]  [<ffffffff817263fc>] ret_from_fork+0x7c/0xb0
    Aug 12 02:30:16 somehostname kernel: [ 6964.779385]  [<ffffffff8108b240>] ? kthread_create_on_node+0x1d0/0x1d0
    Aug 12 02:32:16 somehostname kernel: [ 7084.828482] INFO: task bcache_writebac:29721 blocked for more than 120 seconds.
    Aug 12 02:32:16 somehostname kernel: [ 7084.922987]       Not tainted 3.13.0-24-generic #47-Ubuntu
    Aug 12 02:32:16 somehostname kernel: [ 7084.993239] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
    Aug 12 02:32:16 somehostname kernel: [ 7085.093858] bcache_writebac D ffff8817dfb34440     0 29721      2 0x00000000
    Aug 12 02:32:16 somehostname kernel: [ 7085.093866]  ffff882fafe57eb8 0000000000000002 ffff882fb100afe0 ffff882fafe57fd8
    Aug 12 02:32:16 somehostname kernel: [ 7085.093874]  0000000000014440 0000000000014440 ffff882fb100afe0 ffff882fb7bcac80
    Aug 12 02:32:16 somehostname kernel: [ 7085.093881]  ffff882fb11c0000 ffffffffa03f5840 0000000000000000 0000000000000000
    Aug 12 02:32:16 somehostname kernel: [ 7085.093897] Call Trace:
    Aug 12 02:32:16 somehostname kernel: [ 7085.093918]  [<ffffffffa03f5840>] ? read_dirty+0x400/0x400 [bcache]
    Aug 12 02:32:16 somehostname kernel: [ 7085.093928]  [<ffffffff81719ee9>] schedule+0x29/0x70
    Aug 12 02:32:16 somehostname kernel: [ 7085.093944]  [<ffffffff8108b2ed>] kthread+0xad/0xf0
    Aug 12 02:32:16 somehostname kernel: [ 7085.093950]  [<ffffffff8108b240>] ? kthread_create_on_node+0x1d0/0x1d0
    Aug 12 02:32:16 somehostname kernel: [ 7085.093959]  [<ffffffff817263fc>] ret_from_fork+0x7c/0xb0
    Aug 12 02:32:16 somehostname kernel: [ 7085.093964]  [<ffffffff8108b240>] ? kthread_create_on_node+0x1d0/0x1d0

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: linux-image-3.13.0-33-generic 3.13.0-33.58
ProcVersionSignature: Ubuntu 3.13.0-33.58-generic 3.13.11.4
Uname: Linux 3.13.0-33-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.3
Architecture: amd64
AudioDevicesInUse:
 USER        PID ACCESS COMMAND
 /dev/snd/controlC1:  nick       2424 F.... pulseaudio
 /dev/snd/controlC0:  nick       2424 F.... pulseaudio
CurrentDesktop: Unity
Date: Fri Aug 15 11:08:19 2014
HibernationDevice: RESUME=UUID=dae1fd50-2dc0-48b2-bd93-5823478772b6
InstallationDate: Installed on 2014-05-29 (77 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Release amd64 (20140417)
MachineType: LENOVO 20ALCTO1WW
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.13.0-33-generic.efi.signed root=/dev/mapper/ubuntu--vg-root ro quiet splash vt.handoff=7
RelatedPackageVersions:
 linux-restricted-modules-3.13.0-33-generic N/A
 linux-backports-modules-3.13.0-33-generic  N/A
 linux-firmware                             1.127.5
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 02/06/2014
dmi.bios.vendor: LENOVO
dmi.bios.version: GIET69WW (2.19 )
dmi.board.asset.tag: Not Available
dmi.board.name: 20ALCTO1WW
dmi.board.vendor: LENOVO
dmi.board.version: 0B98405 STD
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: Not Available
dmi.modalias: dmi:bvnLENOVO:bvrGIET69WW(2.19):bd02/06/2014:svnLENOVO:pn20ALCTO1WW:pvrThinkPadX240:rvnLENOVO:rn20ALCTO1WW:rvr0B98405STD:cvnLENOVO:ct10:cvrNotAvailable:
dmi.product.name: 20ALCTO1WW
dmi.product.version: ThinkPad X240
dmi.sys.vendor: LENOVO

** Affects: linux (Ubuntu)
     Importance: Undecided
         Status: New


** Tags: amd64 apport-bug regression-release trusty

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1357295

Title:
  bcache_writebac blocks and goes silent on trusty kernel
  3.13.0-24-generic #47-Ubuntu

Status in “linux” package in Ubuntu:
  New

Bug description:
  We regularly use bcache to use SSDs as an extra block cache layer for
  performance on large working sets (archive mirrors, large databases,
  etc).  Until now we have had good success with bcache on precise LTS
  with HWE kernels up to Saucy's.

  Unfortunately when we mount a bcache-backed volume on a Trusty server,
  it immediately loses contact with bcache_writebac:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                            
    501 root      20   0       0      0      0 D   0.0  0.0   0:00.00 bcache_writebac                    

  kern.log shows it losing its mind pretty much immediately on startup:

      Aug 12 02:22:15 somehostname kernel: [ 6483.477618] md: bind<sdc>
      Aug 12 02:22:15 somehostname kernel: [ 6483.478599] md: bind<sdd>
      Aug 12 02:22:15 somehostname kernel: [ 6483.482016] md/raid0:md0: md_size is 781443072 sectors.
      Aug 12 02:22:15 somehostname kernel: [ 6483.482024] md: RAID0 configuration for md0 - 1 zone
      Aug 12 02:22:15 somehostname kernel: [ 6483.482026] md: zone0=[sdc/sdd]
      Aug 12 02:22:15 somehostname kernel: [ 6483.482033]       zone-offset=         0KB, device-offset=         0KB, size= 390721536KB
      Aug 12 02:22:15 somehostname kernel: [ 6483.482036] 
      Aug 12 02:22:15 somehostname kernel: [ 6483.482055] md0: detected capacity change from 0 to 400098852864
      Aug 12 02:22:15 somehostname kernel: [ 6483.487644]  md0: unknown partition table
      Aug 12 02:26:55 somehostname kernel: [ 6763.657249] bio: create slab <bio-2> at 2
      Aug 12 02:26:55 somehostname kernel: [ 6763.658018] bcache: run_cache_set() invalidating existing data
      Aug 12 02:26:55 somehostname kernel: [ 6763.695447] bcache: register_cache() registered cache device md0
      Aug 12 02:26:55 somehostname kernel: [ 6763.696262] bcache: register_bdev() registered backing device sdb
      Aug 12 02:26:55 somehostname kernel: [ 6763.697833] bcache: bch_cached_dev_attach() Caching sdb as bcache0 on set 13dcf71e-73ea-49f7-8cd6-fb8182c3cc17
      Aug 12 02:27:24 somehostname kernel: [ 6793.114287] EXT4-fs (bcache0): mounted filesystem with ordered data mode. Opts: (null)
      Aug 12 02:30:16 somehostname kernel: [ 6964.514425] INFO: task bcache_writebac:29721 blocked for more than 120 seconds.
      Aug 12 02:30:16 somehostname kernel: [ 6964.608861]       Not tainted 3.13.0-24-generic #47-Ubuntu
      Aug 12 02:30:16 somehostname kernel: [ 6964.679403] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
      Aug 12 02:30:16 somehostname kernel: [ 6964.779270] bcache_writebac D ffff8817dfb34440     0 29721      2 0x00000000
      Aug 12 02:30:16 somehostname kernel: [ 6964.779278]  ffff882fafe57eb8 0000000000000002 ffff882fb100afe0 ffff882fafe57fd8
      Aug 12 02:30:16 somehostname kernel: [ 6964.779286]  0000000000014440 0000000000014440 ffff882fb100afe0 ffff882fb7bcac80
      Aug 12 02:30:16 somehostname kernel: [ 6964.779293]  ffff882fb11c0000 ffffffffa03f5840 0000000000000000 0000000000000000
      Aug 12 02:30:16 somehostname kernel: [ 6964.779301] Call Trace:
      Aug 12 02:30:16 somehostname kernel: [ 6964.779335]  [<ffffffffa03f5840>] ? read_dirty+0x400/0x400 [bcache]
      Aug 12 02:30:16 somehostname kernel: [ 6964.779346]  [<ffffffff81719ee9>] schedule+0x29/0x70
      Aug 12 02:30:16 somehostname kernel: [ 6964.779356]  [<ffffffff8108b2ed>] kthread+0xad/0xf0
      Aug 12 02:30:16 somehostname kernel: [ 6964.779368]  [<ffffffff8108b240>] ? kthread_create_on_node+0x1d0/0x1d0
      Aug 12 02:30:16 somehostname kernel: [ 6964.779379]  [<ffffffff817263fc>] ret_from_fork+0x7c/0xb0
      Aug 12 02:30:16 somehostname kernel: [ 6964.779385]  [<ffffffff8108b240>] ? kthread_create_on_node+0x1d0/0x1d0
      Aug 12 02:32:16 somehostname kernel: [ 7084.828482] INFO: task bcache_writebac:29721 blocked for more than 120 seconds.
      Aug 12 02:32:16 somehostname kernel: [ 7084.922987]       Not tainted 3.13.0-24-generic #47-Ubuntu
      Aug 12 02:32:16 somehostname kernel: [ 7084.993239] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
      Aug 12 02:32:16 somehostname kernel: [ 7085.093858] bcache_writebac D ffff8817dfb34440     0 29721      2 0x00000000
      Aug 12 02:32:16 somehostname kernel: [ 7085.093866]  ffff882fafe57eb8 0000000000000002 ffff882fb100afe0 ffff882fafe57fd8
      Aug 12 02:32:16 somehostname kernel: [ 7085.093874]  0000000000014440 0000000000014440 ffff882fb100afe0 ffff882fb7bcac80
      Aug 12 02:32:16 somehostname kernel: [ 7085.093881]  ffff882fb11c0000 ffffffffa03f5840 0000000000000000 0000000000000000
      Aug 12 02:32:16 somehostname kernel: [ 7085.093897] Call Trace:
      Aug 12 02:32:16 somehostname kernel: [ 7085.093918]  [<ffffffffa03f5840>] ? read_dirty+0x400/0x400 [bcache]
      Aug 12 02:32:16 somehostname kernel: [ 7085.093928]  [<ffffffff81719ee9>] schedule+0x29/0x70
      Aug 12 02:32:16 somehostname kernel: [ 7085.093944]  [<ffffffff8108b2ed>] kthread+0xad/0xf0
      Aug 12 02:32:16 somehostname kernel: [ 7085.093950]  [<ffffffff8108b240>] ? kthread_create_on_node+0x1d0/0x1d0
      Aug 12 02:32:16 somehostname kernel: [ 7085.093959]  [<ffffffff817263fc>] ret_from_fork+0x7c/0xb0
      Aug 12 02:32:16 somehostname kernel: [ 7085.093964]  [<ffffffff8108b240>] ? kthread_create_on_node+0x1d0/0x1d0

  ProblemType: Bug
  DistroRelease: Ubuntu 14.04
  Package: linux-image-3.13.0-33-generic 3.13.0-33.58
  ProcVersionSignature: Ubuntu 3.13.0-33.58-generic 3.13.11.4
  Uname: Linux 3.13.0-33-generic x86_64
  ApportVersion: 2.14.1-0ubuntu3.3
  Architecture: amd64
  AudioDevicesInUse:
   USER        PID ACCESS COMMAND
   /dev/snd/controlC1:  nick       2424 F.... pulseaudio
   /dev/snd/controlC0:  nick       2424 F.... pulseaudio
  CurrentDesktop: Unity
  Date: Fri Aug 15 11:08:19 2014
  HibernationDevice: RESUME=UUID=dae1fd50-2dc0-48b2-bd93-5823478772b6
  InstallationDate: Installed on 2014-05-29 (77 days ago)
  InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Release amd64 (20140417)
  MachineType: LENOVO 20ALCTO1WW
  ProcFB: 0 inteldrmfb
  ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.13.0-33-generic.efi.signed root=/dev/mapper/ubuntu--vg-root ro quiet splash vt.handoff=7
  RelatedPackageVersions:
   linux-restricted-modules-3.13.0-33-generic N/A
   linux-backports-modules-3.13.0-33-generic  N/A
   linux-firmware                             1.127.5
  SourcePackage: linux
  UpgradeStatus: No upgrade log present (probably fresh install)
  dmi.bios.date: 02/06/2014
  dmi.bios.vendor: LENOVO
  dmi.bios.version: GIET69WW (2.19 )
  dmi.board.asset.tag: Not Available
  dmi.board.name: 20ALCTO1WW
  dmi.board.vendor: LENOVO
  dmi.board.version: 0B98405 STD
  dmi.chassis.asset.tag: No Asset Information
  dmi.chassis.type: 10
  dmi.chassis.vendor: LENOVO
  dmi.chassis.version: Not Available
  dmi.modalias: dmi:bvnLENOVO:bvrGIET69WW(2.19):bd02/06/2014:svnLENOVO:pn20ALCTO1WW:pvrThinkPadX240:rvnLENOVO:rn20ALCTO1WW:rvr0B98405STD:cvnLENOVO:ct10:cvrNotAvailable:
  dmi.product.name: 20ALCTO1WW
  dmi.product.version: ThinkPad X240
  dmi.sys.vendor: LENOVO

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1357295/+subscriptions