kernel-packages team mailing list archive

[Haw Loeung <1357295@xxxxxxxxxxxxxxxxxx>]

Looks like there's a pending patch[1].

I used the fix suggested by Sam Fulcomer[2] to set
/sys/.../bcache/writeback_running to 0 because the cache mod we're using
is writethrough.

| $ cat /sys/block/bcache0/bcache/writeback_running
| 1

| $cat /sys/block/bcache0/bcache/state
| clean

| $ cat /sys/block/bcache0/bcach/cache_mode
| [writethrough] writeback writearound none

Load seems to have dropped from a constant 2 to around 1. I've left it
over night and am not seeing any more call traces being logged.

[1] http://permalink.gmane.org/gmane.linux.kernel.bcache.devel/2499
[2] http://permalink.gmane.org/gmane.linux.kernel.bcache.devel/2492

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1357295

Title:
  bcache_writebac blocks and goes silent on trusty kernel
  3.13.0-24-generic #47-Ubuntu

Status in “linux” package in Ubuntu:
  Confirmed

Bug description:
  We regularly use bcache to use SSDs as an extra block cache layer for
  performance on large working sets (archive mirrors, large databases,
  etc).  Until now we have had good success with bcache on precise LTS
  with HWE kernels up to Saucy's.

  Unfortunately when we mount a bcache-backed volume on a Trusty server,
  it immediately loses contact with bcache_writebac:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                            
    501 root      20   0       0      0      0 D   0.0  0.0   0:00.00 bcache_writebac                    

  kern.log shows it losing its mind pretty much immediately on startup:

      Aug 12 02:22:15 somehostname kernel: [ 6483.477618] md: bind<sdc>
      Aug 12 02:22:15 somehostname kernel: [ 6483.478599] md: bind<sdd>
      Aug 12 02:22:15 somehostname kernel: [ 6483.482016] md/raid0:md0: md_size is 781443072 sectors.
      Aug 12 02:22:15 somehostname kernel: [ 6483.482024] md: RAID0 configuration for md0 - 1 zone
      Aug 12 02:22:15 somehostname kernel: [ 6483.482026] md: zone0=[sdc/sdd]
      Aug 12 02:22:15 somehostname kernel: [ 6483.482033]       zone-offset=         0KB, device-offset=         0KB, size= 390721536KB
      Aug 12 02:22:15 somehostname kernel: [ 6483.482036] 
      Aug 12 02:22:15 somehostname kernel: [ 6483.482055] md0: detected capacity change from 0 to 400098852864
      Aug 12 02:22:15 somehostname kernel: [ 6483.487644]  md0: unknown partition table
      Aug 12 02:26:55 somehostname kernel: [ 6763.657249] bio: create slab <bio-2> at 2
      Aug 12 02:26:55 somehostname kernel: [ 6763.658018] bcache: run_cache_set() invalidating existing data
      Aug 12 02:26:55 somehostname kernel: [ 6763.695447] bcache: register_cache() registered cache device md0
      Aug 12 02:26:55 somehostname kernel: [ 6763.696262] bcache: register_bdev() registered backing device sdb
      Aug 12 02:26:55 somehostname kernel: [ 6763.697833] bcache: bch_cached_dev_attach() Caching sdb as bcache0 on set 13dcf71e-73ea-49f7-8cd6-fb8182c3cc17
      Aug 12 02:27:24 somehostname kernel: [ 6793.114287] EXT4-fs (bcache0): mounted filesystem with ordered data mode. Opts: (null)
      Aug 12 02:30:16 somehostname kernel: [ 6964.514425] INFO: task bcache_writebac:29721 blocked for more than 120 seconds.
      Aug 12 02:30:16 somehostname kernel: [ 6964.608861]       Not tainted 3.13.0-24-generic #47-Ubuntu
      Aug 12 02:30:16 somehostname kernel: [ 6964.679403] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
      Aug 12 02:30:16 somehostname kernel: [ 6964.779270] bcache_writebac D ffff8817dfb34440     0 29721      2 0x00000000
      Aug 12 02:30:16 somehostname kernel: [ 6964.779278]  ffff882fafe57eb8 0000000000000002 ffff882fb100afe0 ffff882fafe57fd8
      Aug 12 02:30:16 somehostname kernel: [ 6964.779286]  0000000000014440 0000000000014440 ffff882fb100afe0 ffff882fb7bcac80
      Aug 12 02:30:16 somehostname kernel: [ 6964.779293]  ffff882fb11c0000 ffffffffa03f5840 0000000000000000 0000000000000000
      Aug 12 02:30:16 somehostname kernel: [ 6964.779301] Call Trace:
      Aug 12 02:30:16 somehostname kernel: [ 6964.779335]  [<ffffffffa03f5840>] ? read_dirty+0x400/0x400 [bcache]
      Aug 12 02:30:16 somehostname kernel: [ 6964.779346]  [<ffffffff81719ee9>] schedule+0x29/0x70
      Aug 12 02:30:16 somehostname kernel: [ 6964.779356]  [<ffffffff8108b2ed>] kthread+0xad/0xf0
      Aug 12 02:30:16 somehostname kernel: [ 6964.779368]  [<ffffffff8108b240>] ? kthread_create_on_node+0x1d0/0x1d0
      Aug 12 02:30:16 somehostname kernel: [ 6964.779379]  [<ffffffff817263fc>] ret_from_fork+0x7c/0xb0
      Aug 12 02:30:16 somehostname kernel: [ 6964.779385]  [<ffffffff8108b240>] ? kthread_create_on_node+0x1d0/0x1d0
      Aug 12 02:32:16 somehostname kernel: [ 7084.828482] INFO: task bcache_writebac:29721 blocked for more than 120 seconds.
      Aug 12 02:32:16 somehostname kernel: [ 7084.922987]       Not tainted 3.13.0-24-generic #47-Ubuntu
      Aug 12 02:32:16 somehostname kernel: [ 7084.993239] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
      Aug 12 02:32:16 somehostname kernel: [ 7085.093858] bcache_writebac D ffff8817dfb34440     0 29721      2 0x00000000
      Aug 12 02:32:16 somehostname kernel: [ 7085.093866]  ffff882fafe57eb8 0000000000000002 ffff882fb100afe0 ffff882fafe57fd8
      Aug 12 02:32:16 somehostname kernel: [ 7085.093874]  0000000000014440 0000000000014440 ffff882fb100afe0 ffff882fb7bcac80
      Aug 12 02:32:16 somehostname kernel: [ 7085.093881]  ffff882fb11c0000 ffffffffa03f5840 0000000000000000 0000000000000000
      Aug 12 02:32:16 somehostname kernel: [ 7085.093897] Call Trace:
      Aug 12 02:32:16 somehostname kernel: [ 7085.093918]  [<ffffffffa03f5840>] ? read_dirty+0x400/0x400 [bcache]
      Aug 12 02:32:16 somehostname kernel: [ 7085.093928]  [<ffffffff81719ee9>] schedule+0x29/0x70
      Aug 12 02:32:16 somehostname kernel: [ 7085.093944]  [<ffffffff8108b2ed>] kthread+0xad/0xf0
      Aug 12 02:32:16 somehostname kernel: [ 7085.093950]  [<ffffffff8108b240>] ? kthread_create_on_node+0x1d0/0x1d0
      Aug 12 02:32:16 somehostname kernel: [ 7085.093959]  [<ffffffff817263fc>] ret_from_fork+0x7c/0xb0
      Aug 12 02:32:16 somehostname kernel: [ 7085.093964]  [<ffffffff8108b240>] ? kthread_create_on_node+0x1d0/0x1d0

  ProblemType: Bug
  DistroRelease: Ubuntu 14.04
  Package: linux-image-3.13.0-33-generic 3.13.0-33.58
  ProcVersionSignature: Ubuntu 3.13.0-33.58-generic 3.13.11.4
  Uname: Linux 3.13.0-33-generic x86_64
  ApportVersion: 2.14.1-0ubuntu3.3
  Architecture: amd64
  AudioDevicesInUse:
   USER        PID ACCESS COMMAND
   /dev/snd/controlC1:  nick       2424 F.... pulseaudio
   /dev/snd/controlC0:  nick       2424 F.... pulseaudio
  CurrentDesktop: Unity
  Date: Fri Aug 15 11:08:19 2014
  HibernationDevice: RESUME=UUID=dae1fd50-2dc0-48b2-bd93-5823478772b6
  InstallationDate: Installed on 2014-05-29 (77 days ago)
  InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Release amd64 (20140417)
  MachineType: LENOVO 20ALCTO1WW
  ProcFB: 0 inteldrmfb
  ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.13.0-33-generic.efi.signed root=/dev/mapper/ubuntu--vg-root ro quiet splash vt.handoff=7
  RelatedPackageVersions:
   linux-restricted-modules-3.13.0-33-generic N/A
   linux-backports-modules-3.13.0-33-generic  N/A
   linux-firmware                             1.127.5
  SourcePackage: linux
  UpgradeStatus: No upgrade log present (probably fresh install)
  dmi.bios.date: 02/06/2014
  dmi.bios.vendor: LENOVO
  dmi.bios.version: GIET69WW (2.19 )
  dmi.board.asset.tag: Not Available
  dmi.board.name: 20ALCTO1WW
  dmi.board.vendor: LENOVO
  dmi.board.version: 0B98405 STD
  dmi.chassis.asset.tag: No Asset Information
  dmi.chassis.type: 10
  dmi.chassis.vendor: LENOVO
  dmi.chassis.version: Not Available
  dmi.modalias: dmi:bvnLENOVO:bvrGIET69WW(2.19):bd02/06/2014:svnLENOVO:pn20ALCTO1WW:pvrThinkPadX240:rvnLENOVO:rn20ALCTO1WW:rvr0B98405STD:cvnLENOVO:ct10:cvrNotAvailable:
  dmi.product.name: 20ALCTO1WW
  dmi.product.version: ThinkPad X240
  dmi.sys.vendor: LENOVO

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1357295/+subscriptions