kernel-packages team mailing list archive

[Gavin Guo <gavin.guo@xxxxxxxxxxxxx>]

** Description changed:

  We are trying to debug the kernel using KASAN and we found that when a
  VM is booting in our cloud, on the virtualised kernel, there is a use-
  after-free access that should not be there.
  
  The failing VM was running on a host with kernel 3.13.0-66-generic
  (trusty). Hosts' qemu version: 1:2.2+dfsg-5expubuntu9.3~cloud0. Hosts'
  seabios: 1.7.5-1ubuntu1~cloud0
  
  The flavour of this VM is 4 CPUs, 8G RAM, 80G of root disk, 0 G swap and
  0 G ephemeral disk.
  
  Here is the trace from KASAN (from the VM):
  
  The error message can be observed in the dmesg when the guest VM booted
- with v3.13.0-65 with KASAN enabled.
+ with v3.13.0-65 with KASAN enabled and "slub_debug=PU,kmalloc-32" in
+ kernel command line.
  
  ==================================================================
- BUG: KASan: use after free in blk_mq_register_disk+0x193/0x260 at addr ffff8801ec247400
+ BUG: KASan: out of bounds access in blk_mq_register_disk+0x193/0x260 at addr ffff8801f43f4d90
  Read of size 8 by task swapper/0/1
  =============================================================================
  BUG kmalloc-32 (Not tainted): kasan: bad access detected
  -----------------------------------------------------------------------------
  
  Disabling lock debugging due to kernel taint
- INFO: Slab 0xffffea0007b091c0 objects=128 used=128 fp=0x          (null) flags=0x2ffff0000000080
- INFO: Object 0xffff8801ec247400 @offset=1024 fp=0xffff8801ec247420
- 
- Bytes b4 ffff8801ec2473f0: 00 ac 71 ef 01 88 ff ff 00 ac 79 ef 01 88 ff ff  ..q.......y.....
- Object ffff8801ec247400: 20 74 24 ec 01 88 ff ff 2f 76 69 72 74 75 61 6c   t$...../virtual
- Object ffff8801ec247410: 2f 62 64 69 2f 32 35 33 3a 30 00 00 00 00 00 00  /bdi/253:0......
+ INFO: Allocated in blk_mq_init_hw_queues+0x778/0x920 age=5 cpu=1 pid=1
+         __slab_alloc+0x4f8/0x560
+         __kmalloc_node+0xad/0x310
+         blk_mq_init_hw_queues+0x778/0x920
+         blk_mq_init_queue+0x5f7/0x6c0
+         virtblk_probe+0x207/0x980
+         virtio_dev_probe+0x1be/0x280
+         driver_probe_device+0xe2/0x5c0
+         __driver_attach+0xc3/0xd0
+         bus_for_each_dev+0x95/0xe0
+         driver_attach+0x2b/0x30
+         bus_add_driver+0x268/0x360
+         driver_register+0xd3/0x1a0
+         register_virtio_driver+0x3c/0x60
+         init+0x53/0x80
+         do_one_initcall+0xda/0x1a0
+         kernel_init_freeable+0x1eb/0x27e
+ INFO: Freed in kzfree+0x2d/0x40 age=13 cpu=0 pid=8
+         __slab_free+0x2ab/0x3f0
+         kfree+0x161/0x170
+         kzfree+0x2d/0x40
+         aa_free_task_context+0x5d/0xa0
+         apparmor_cred_free+0x24/0x40
+         security_cred_free+0x2b/0x30
+         put_cred_rcu+0x38/0x140
+         rcu_nocb_kthread+0x25a/0x410
+         kthread+0x101/0x120
+         ret_from_fork+0x58/0x90
+ INFO: Slab 0xffffea0007d0fd00 objects=23 used=21 fp=0xffff8801f43f52d0 flags=0x2ffff0000004080
+ INFO: Object 0xffff8801f43f4d70 @offset=3440 fp=0xffff8801f43f5830
+ Bytes b4 ffff8801f43f4d60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
+ Object ffff8801f43f4d70: 00 ac 61 f7 01 88 ff ff 00 ac 69 f7 01 88 ff ff  ..a.......i.....
+ Object ffff8801f43f4d80: 00 ac 71 f7 01 88 ff ff 00 ac 79 f7 01 88 ff ff  ..q.......y.....
  CPU: 0 PID: 1 Comm: swapper/0 Tainted: G    B         3.13.0-65-generic #105
  Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.7.5-20150310_111955-batsu 04/01/2014
-  ffffea0007b091c0 ffff8801ec0cb9a8 ffffffff81a6ce35 ffff8801ef001c00
-  ffff8801ec0cb9d8 ffffffff81244aed ffff8801ef001c00 ffffea0007b091c0
-  ffff8801ec247400 ffff8801ef79ac98 ffff8801ec0cba00 ffffffff8124ac36
+  ffffea0007d0fd00 ffff8801f40cf9a8 ffffffff81a6ce35 ffff8801f7001c00
+  ffff8801f40cf9d8 ffffffff81244aed ffff8801f7001c00 ffffea0007d0fd00
+  ffff8801f43f4d70 ffff8801f779ac98 ffff8801f40cfa00 ffffffff8124ac36
  Call Trace:
-  [<ffffffff81a6ce35>] dump_stack+0x45/0x56
-  [<ffffffff81244aed>] print_trailer+0xfd/0x170
-  [<ffffffff8124ac36>] object_err+0x36/0x40
-  [<ffffffff8124cbf9>] kasan_report_error+0x1e9/0x3a0
-  [<ffffffff81319427>] ? sysfs_get+0x17/0x50
-  [<ffffffff814dee6b>] ? kobject_add_internal+0x29b/0x4a0
-  [<ffffffff8124d260>] kasan_report+0x40/0x50
-  [<ffffffff81696f00>] ? dev_printk_emit+0x20/0x40
-  [<ffffffff814ae7c3>] ? blk_mq_register_disk+0x193/0x260
-  [<ffffffff8124bee9>] __asan_load8+0x69/0xa0
-  [<ffffffff814ae7c3>] blk_mq_register_disk+0x193/0x260
-  [<ffffffff814a1572>] blk_register_queue+0xd2/0x170
-  [<ffffffff814b24cf>] add_disk+0x31f/0x720
-  [<ffffffff816ced9a>] virtblk_probe+0x58a/0x980
-  [<ffffffff816cd4c0>] ? virtblk_restore+0x100/0x100
-  [<ffffffff81601b8e>] virtio_dev_probe+0x1be/0x280
-  [<ffffffff8169d620>] ? __device_attach+0x70/0x70
-  [<ffffffff8169d0d2>] driver_probe_device+0xe2/0x5c0
-  [<ffffffff8169d620>] ? __device_attach+0x70/0x70
-  [<ffffffff8169d6e3>] __driver_attach+0xc3/0xd0
-  [<ffffffff8169a355>] bus_for_each_dev+0x95/0xe0
-  [<ffffffff8169c89b>] driver_attach+0x2b/0x30
-  [<ffffffff8169c298>] bus_add_driver+0x268/0x360
-  [<ffffffff8169dfe3>] driver_register+0xd3/0x1a0
-  [<ffffffff8218e4b9>] ? loop_init+0x14b/0x14b
-  [<ffffffff8160213c>] register_virtio_driver+0x3c/0x60
-  [<ffffffff8218e50c>] init+0x53/0x80
-  [<ffffffff8100212a>] do_one_initcall+0xda/0x1a0
-  [<ffffffff8213816b>] kernel_init_freeable+0x1eb/0x27e
-  [<ffffffff81a5bcd0>] ? rest_init+0x80/0x80
-  [<ffffffff81a5bcde>] kernel_init+0xe/0x130
-  [<ffffffff81a83028>] ret_from_fork+0x58/0x90
-  [<ffffffff81a5bcd0>] ? rest_init+0x80/0x80
+  [<ffffffff81a6ce35>] dump_stack+0x45/0x56
+  [<ffffffff81244aed>] print_trailer+0xfd/0x170
+  [<ffffffff8124ac36>] object_err+0x36/0x40
+  [<ffffffff8124cbf9>] kasan_report_error+0x1e9/0x3a0
+  [<ffffffff81319427>] ? sysfs_get+0x17/0x50
+  [<ffffffff814dee6b>] ? kobject_add_internal+0x29b/0x4a0
+  [<ffffffff8124d260>] kasan_report+0x40/0x50
+  [<ffffffff81696f00>] ? dev_printk_emit+0x20/0x40
+  [<ffffffff814ae7c3>] ? blk_mq_register_disk+0x193/0x260
+  [<ffffffff8124bee9>] __asan_load8+0x69/0xa0
+  [<ffffffff814ae7c3>] blk_mq_register_disk+0x193/0x260
+  [<ffffffff814a1572>] blk_register_queue+0xd2/0x170
+  [<ffffffff814b24cf>] add_disk+0x31f/0x720
+  [<ffffffff816ced9a>] virtblk_probe+0x58a/0x980
+  [<ffffffff816cd4c0>] ? virtblk_restore+0x100/0x100
+  [<ffffffff81601b8e>] virtio_dev_probe+0x1be/0x280
+  [<ffffffff8169d620>] ? __device_attach+0x70/0x70
+  [<ffffffff8169d0d2>] driver_probe_device+0xe2/0x5c0
+  [<ffffffff8169d620>] ? __device_attach+0x70/0x70
+  [<ffffffff8169d6e3>] __driver_attach+0xc3/0xd0
+  [<ffffffff8169a355>] bus_for_each_dev+0x95/0xe0
+  [<ffffffff8169c89b>] driver_attach+0x2b/0x30
+  [<ffffffff8169c298>] bus_add_driver+0x268/0x360
+  [<ffffffff8169dfe3>] driver_register+0xd3/0x1a0
+  [<ffffffff8218e4b9>] ? loop_init+0x14b/0x14b
+  [<ffffffff8160213c>] register_virtio_driver+0x3c/0x60
+  [<ffffffff8218e50c>] init+0x53/0x80
+  [<ffffffff8100212a>] do_one_initcall+0xda/0x1a0
+  [<ffffffff8213816b>] kernel_init_freeable+0x1eb/0x27e
+  [<ffffffff81a5bcd0>] ? rest_init+0x80/0x80
+  [<ffffffff81a5bcde>] kernel_init+0xe/0x130
+  [<ffffffff81a83028>] ret_from_fork+0x58/0x90
+  [<ffffffff81a5bcd0>] ? rest_init+0x80/0x80
  Memory state around the buggy address:
-  ffff8801ec247300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
-  ffff8801ec247380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
- >ffff8801ec247400: fb fb fb fb fc fc fc fc fc fc fc fc fc fc fc fc
-                    ^
-  ffff8801ec247480: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
-  ffff8801ec247500: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
+  ffff8801f43f4c80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
+  ffff8801f43f4d00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc 00 00
+ >ffff8801f43f4d80: 00 00 fc fc fc fc fc fc fc fc fc fc fc fc fc fc
+                          ^
+  ffff8801f43f4e00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
+  ffff8801f43f4e80: fc fc fc fc fc fc fc fc fc 00 00 00 00 fc fc fc
  ==================================================================

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1534054

Title:
  use-after-free found by KASAN in blk_mq_register_disk

Status in linux package in Ubuntu:
  Incomplete

Bug description:
  We are trying to debug the kernel using KASAN and we found that when a
  VM is booting in our cloud, on the virtualised kernel, there is a use-
  after-free access that should not be there.

  The failing VM was running on a host with kernel 3.13.0-66-generic
  (trusty). Hosts' qemu version: 1:2.2+dfsg-5expubuntu9.3~cloud0. Hosts'
  seabios: 1.7.5-1ubuntu1~cloud0

  The flavour of this VM is 4 CPUs, 8G RAM, 80G of root disk, 0 G swap
  and 0 G ephemeral disk.

  Here is the trace from KASAN (from the VM):

  The error message can be observed in the dmesg when the guest VM
  booted with v3.13.0-65 with KASAN enabled and
  "slub_debug=PU,kmalloc-32" in kernel command line.

  ==================================================================
  BUG: KASan: out of bounds access in blk_mq_register_disk+0x193/0x260 at addr ffff8801f43f4d90
  Read of size 8 by task swapper/0/1
  =============================================================================
  BUG kmalloc-32 (Not tainted): kasan: bad access detected
  -----------------------------------------------------------------------------

  Disabling lock debugging due to kernel taint
  INFO: Allocated in blk_mq_init_hw_queues+0x778/0x920 age=5 cpu=1 pid=1
          __slab_alloc+0x4f8/0x560
          __kmalloc_node+0xad/0x310
          blk_mq_init_hw_queues+0x778/0x920
          blk_mq_init_queue+0x5f7/0x6c0
          virtblk_probe+0x207/0x980
          virtio_dev_probe+0x1be/0x280
          driver_probe_device+0xe2/0x5c0
          __driver_attach+0xc3/0xd0
          bus_for_each_dev+0x95/0xe0
          driver_attach+0x2b/0x30
          bus_add_driver+0x268/0x360
          driver_register+0xd3/0x1a0
          register_virtio_driver+0x3c/0x60
          init+0x53/0x80
          do_one_initcall+0xda/0x1a0
          kernel_init_freeable+0x1eb/0x27e
  INFO: Freed in kzfree+0x2d/0x40 age=13 cpu=0 pid=8
          __slab_free+0x2ab/0x3f0
          kfree+0x161/0x170
          kzfree+0x2d/0x40
          aa_free_task_context+0x5d/0xa0
          apparmor_cred_free+0x24/0x40
          security_cred_free+0x2b/0x30
          put_cred_rcu+0x38/0x140
          rcu_nocb_kthread+0x25a/0x410
          kthread+0x101/0x120
          ret_from_fork+0x58/0x90
  INFO: Slab 0xffffea0007d0fd00 objects=23 used=21 fp=0xffff8801f43f52d0 flags=0x2ffff0000004080
  INFO: Object 0xffff8801f43f4d70 @offset=3440 fp=0xffff8801f43f5830
  Bytes b4 ffff8801f43f4d60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  Object ffff8801f43f4d70: 00 ac 61 f7 01 88 ff ff 00 ac 69 f7 01 88 ff ff  ..a.......i.....
  Object ffff8801f43f4d80: 00 ac 71 f7 01 88 ff ff 00 ac 79 f7 01 88 ff ff  ..q.......y.....
  CPU: 0 PID: 1 Comm: swapper/0 Tainted: G    B         3.13.0-65-generic #105
  Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.7.5-20150310_111955-batsu 04/01/2014
   ffffea0007d0fd00 ffff8801f40cf9a8 ffffffff81a6ce35 ffff8801f7001c00
   ffff8801f40cf9d8 ffffffff81244aed ffff8801f7001c00 ffffea0007d0fd00
   ffff8801f43f4d70 ffff8801f779ac98 ffff8801f40cfa00 ffffffff8124ac36
  Call Trace:
   [<ffffffff81a6ce35>] dump_stack+0x45/0x56
   [<ffffffff81244aed>] print_trailer+0xfd/0x170
   [<ffffffff8124ac36>] object_err+0x36/0x40
   [<ffffffff8124cbf9>] kasan_report_error+0x1e9/0x3a0
   [<ffffffff81319427>] ? sysfs_get+0x17/0x50
   [<ffffffff814dee6b>] ? kobject_add_internal+0x29b/0x4a0
   [<ffffffff8124d260>] kasan_report+0x40/0x50
   [<ffffffff81696f00>] ? dev_printk_emit+0x20/0x40
   [<ffffffff814ae7c3>] ? blk_mq_register_disk+0x193/0x260
   [<ffffffff8124bee9>] __asan_load8+0x69/0xa0
   [<ffffffff814ae7c3>] blk_mq_register_disk+0x193/0x260
   [<ffffffff814a1572>] blk_register_queue+0xd2/0x170
   [<ffffffff814b24cf>] add_disk+0x31f/0x720
   [<ffffffff816ced9a>] virtblk_probe+0x58a/0x980
   [<ffffffff816cd4c0>] ? virtblk_restore+0x100/0x100
   [<ffffffff81601b8e>] virtio_dev_probe+0x1be/0x280
   [<ffffffff8169d620>] ? __device_attach+0x70/0x70
   [<ffffffff8169d0d2>] driver_probe_device+0xe2/0x5c0
   [<ffffffff8169d620>] ? __device_attach+0x70/0x70
   [<ffffffff8169d6e3>] __driver_attach+0xc3/0xd0
   [<ffffffff8169a355>] bus_for_each_dev+0x95/0xe0
   [<ffffffff8169c89b>] driver_attach+0x2b/0x30
   [<ffffffff8169c298>] bus_add_driver+0x268/0x360
   [<ffffffff8169dfe3>] driver_register+0xd3/0x1a0
   [<ffffffff8218e4b9>] ? loop_init+0x14b/0x14b
   [<ffffffff8160213c>] register_virtio_driver+0x3c/0x60
   [<ffffffff8218e50c>] init+0x53/0x80
   [<ffffffff8100212a>] do_one_initcall+0xda/0x1a0
   [<ffffffff8213816b>] kernel_init_freeable+0x1eb/0x27e
   [<ffffffff81a5bcd0>] ? rest_init+0x80/0x80
   [<ffffffff81a5bcde>] kernel_init+0xe/0x130
   [<ffffffff81a83028>] ret_from_fork+0x58/0x90
   [<ffffffff81a5bcd0>] ? rest_init+0x80/0x80
  Memory state around the buggy address:
   ffff8801f43f4c80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
   ffff8801f43f4d00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc 00 00
  >ffff8801f43f4d80: 00 00 fc fc fc fc fc fc fc fc fc fc fc fc fc fc
                           ^
   ffff8801f43f4e00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
   ffff8801f43f4e80: fc fc fc fc fc fc fc fc fc 00 00 00 00 fc fc fc
  ==================================================================

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1534054/+subscriptions