kernel-packages team mailing list archive

[Mike Lowe <1216444@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

On multiple machines with vm's using vhost-net this bug takes out the
guest network interface under load.  The vm is only able to see
broadcast traffic after this happens.  Started happening immediately
after upgrading from linux-image-3.8.0-27-generic to linux-
image-3.8.0-29-generic.  This changelog entry from linux_3.8.0-28.41
seems particularly relevant:

  * vhost-net: fix use-after-free in vhost_net_flush
    - LP: #1202992
    - CVE-2013-4127

Also seems to be giving the fedora folks fits as well:
https://bugzilla.redhat.com/show_bug.cgi?id=975065

Aug 24 20:00:55 gwbvm4 kernel: [277318.536525] BUG: unable to handle kernel NULL pointer dereference at 00000000000001ea
Aug 24 20:00:55 gwbvm4 kernel: [277318.537027] IP: [<ffffffff8113c1a5>] put_page+0x5/0x40
Aug 24 20:00:55 gwbvm4 kernel: [277318.537359] PGD 0
Aug 24 20:00:55 gwbvm4 kernel: [277318.537505] Oops: 0000 [#1] SMP
Aug 24 20:00:55 gwbvm4 kernel: [277318.537716] Modules linked in: xt_recent(F) nfnetlink_log(F) nfnetlink(F) vhost_net macvtap(F) macvlan(F) brcompat(OF) openvswitch(OF) mptctl(F) mptbase(F) ipmi_devintf ipmi_si ipmi_msghandler ebtable_nat(F) ebtables(F) ipt_MASQUERADE(F) iptable_nat(F) nf_nat_ipv4(F) xt_CHECKSUM(F) iptable_mangle(F) ib_iser rdma_cm ib_addr iw_cm ib_cm ib_sa ib_mad ib_core iscsi_tcp(F) libiscsi_tcp(F) libiscsi(F) scsi_transport_iscsi(F) stp(F) llc(F) ip6t_REJECT(F) xt_hl(F) ip6t_rt(F) nf_conntrack_ipv6(F) nf_defrag_ipv6(F) ipt_REJECT(F) xt_comment(F) xt_limit(F) xt_tcpudp(F) vesafb(F) xt_addrtype(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) xt_state(F) ip6table_filter(F) ip6_tables(F) nf_conntrack_netbios_ns(F) nf_conntrack_broadcast(F) nf_nat_ftp(F) nf_nat(F) nf_conntrack_ftp(F) nf_conntrack(F) iptable_filter(F) ip_tables(F) coretemp x_tables(F) kvm_intel kvm ghash_clmulni_intel(F) aesni_intel(F) aes_x86_64(F) xts(F) lrw(F) gf128mul(F) ablk_helper(F) cryptd(F) gpio_ich lpc_ich microcode(F) serio_r
Aug 24 20:00:55 gwbvm4 kernel: aw(F) i7core_edac mac_hid edac_core lp(F) parport(F) btrfs(F) zlib_deflate(F) libcrc32c(F) ahci(F) libahci(F) igb cxgb3 dca ptp hpsa mdio pps_core [last unloaded: bridge]
Aug 24 20:00:55 gwbvm4 kernel: [277318.544745] CPU 0
Aug 24 20:00:55 gwbvm4 kernel: [277318.544866] Pid: 5489, comm: vhost-5488 Tainted: GF         IO 3.8.0-29-generic #42-Ubuntu HP ProLiant DL160 G6
Aug 24 20:00:55 gwbvm4 kernel: [277318.545560] RIP: 0010:[<ffffffff8113c1a5>]  [<ffffffff8113c1a5>] put_page+0x5/0x40
Aug 24 20:00:55 gwbvm4 kernel: [277318.546034] RSP: 0018:ffff8817ccbc1c78  EFLAGS: 00010202
Aug 24 20:00:55 gwbvm4 kernel: [277318.546356] RAX: ffff8809728a1ac0 RBX: 0000000000000012 RCX: ffff8809728a1ac0
Aug 24 20:00:55 gwbvm4 kernel: [277318.569029] RDX: 0000000000000140 RSI: ffff8809728a1ac0 RDI: 00000000000001ea
Aug 24 20:00:55 gwbvm4 kernel: [277318.592195] RBP: ffff8817ccbc1c90 R08: ffff880970704518 R09: 0000000000000010
Aug 24 20:00:55 gwbvm4 kernel: [277318.615537] R10: 0000000000000001 R11: 0000000000000007 R12: ffff881645ca5100
Aug 24 20:00:55 gwbvm4 kernel: [277318.639881] R13: ffffffff814dfa35 R14: 000000000000000c R15: ffff881645ca5100
Aug 24 20:00:55 gwbvm4 kernel: [277318.664386] FS:  0000000000000000(0000) GS:ffff880c0fc00000(0000) knlGS:0000000000000000
Aug 24 20:00:55 gwbvm4 kernel: [277318.689030] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 24 20:00:55 gwbvm4 kernel: [277318.701226] CR2: 00000000000001ea CR3: 000000069ca0b000 CR4: 00000000000027e0
Aug 24 20:00:55 gwbvm4 kernel: [277318.725240] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 24 20:00:55 gwbvm4 kernel: [277318.749120] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 24 20:00:55 gwbvm4 kernel: [277318.773018] Process vhost-5488 (pid: 5489, threadinfo ffff8817ccbc0000, task ffff8817f6a80000)
Aug 24 20:00:55 gwbvm4 kernel: [277318.796899] Stack:
Aug 24 20:00:55 gwbvm4 kernel: [277318.808406]  ffffffff815bcebf ffff881645ca5100 ffff881645ca5100 ffff8817ccbc1ca8
Aug 24 20:00:55 gwbvm4 kernel: [277318.831675]  ffffffff815bcf5a ffff8809707043d8 ffff8817ccbc1cd0 ffffffff815bd012
Aug 24 20:00:55 gwbvm4 kernel: [277318.855551]  ffff8809707043d8 000000000000f4ee ffff880a2c048800 ffff8817ccbc1d58
Aug 24 20:00:55 gwbvm4 kernel: [277318.879408] Call Trace:
Aug 24 20:00:55 gwbvm4 kernel: [277318.891138]  [<ffffffff815bcebf>] ? skb_release_data+0x8f/0x110
Aug 24 20:00:55 gwbvm4 kernel: [277318.903096]  [<ffffffff815bcf5a>] __kfree_skb+0x1a/0xa0
Aug 24 20:00:55 gwbvm4 kernel: [277318.914622]  [<ffffffff815bd012>] kfree_skb+0x32/0x90
Aug 24 20:00:55 gwbvm4 kernel: [277318.925947]  [<ffffffff814dfa35>] tun_get_user+0x5f5/0x720
Aug 24 20:00:55 gwbvm4 kernel: [277318.937089]  [<ffffffff814dfbb7>] tun_sendmsg+0x57/0x80
Aug 24 20:00:55 gwbvm4 kernel: [277318.947987]  [<ffffffffa0435656>] handle_tx+0x266/0x580 [vhost_net]
Aug 24 20:00:55 gwbvm4 kernel: [277318.958700]  [<ffffffffa04359a5>] handle_tx_kick+0x15/0x20 [vhost_net]
Aug 24 20:00:55 gwbvm4 kernel: [277318.969222]  [<ffffffffa043295f>] vhost_worker+0xff/0x1b0 [vhost_net]
Aug 24 20:00:55 gwbvm4 kernel: [277318.979546]  [<ffffffffa0432860>] ? vhost_work_flush+0x130/0x130 [vhost_net]
Aug 24 20:00:55 gwbvm4 kernel: [277318.989849]  [<ffffffff8107d590>] kthread+0xc0/0xd0
Aug 24 20:00:55 gwbvm4 kernel: [277319.000170]  [<ffffffff8107d4d0>] ? kthread_create_on_node+0x120/0x120
Aug 24 20:00:55 gwbvm4 kernel: [277319.010475]  [<ffffffff816d556c>] ret_from_fork+0x7c/0xb0
Aug 24 20:00:55 gwbvm4 kernel: [277319.020339]  [<ffffffff8107d4d0>] ? kthread_create_on_node+0x120/0x120
Aug 24 20:00:55 gwbvm4 kernel: [277319.030361] Code: fc 00 00 00 00 e8 ac fe ff ff 48 63 45 fc 65 48 01 04 25 78 08 01 00 c9 c3 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 <48> f7 07 00 c0 00 00 55 48 89 e5 75 15 f0 ff 4f 1c 0f 94 c0 84
Aug 24 20:00:55 gwbvm4 kernel: [277319.060251] RIP  [<ffffffff8113c1a5>] put_page+0x5/0x40
Aug 24 20:00:55 gwbvm4 kernel: [277319.069747]  RSP <ffff8817ccbc1c78>
Aug 24 20:00:55 gwbvm4 kernel: [277319.078868] CR2: 00000000000001ea
Aug 24 20:00:55 gwbvm4 kernel: [277319.102160] ---[ end trace def21f8b2fed77aa ]---

** Affects: linux (Ubuntu)
     Importance: Undecided
         Status: New

** Attachment added: "version.log"
   https://bugs.launchpad.net/bugs/1216444/+attachment/3786560/+files/version.log

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1216444

Title:
  Vhost-net made unstable by linux_3.8.0-28.41

Status in “linux” package in Ubuntu:
  New

Bug description:
  On multiple machines with vm's using vhost-net this bug takes out the
  guest network interface under load.  The vm is only able to see
  broadcast traffic after this happens.  Started happening immediately
  after upgrading from linux-image-3.8.0-27-generic to linux-
  image-3.8.0-29-generic.  This changelog entry from linux_3.8.0-28.41
  seems particularly relevant:

    * vhost-net: fix use-after-free in vhost_net_flush
      - LP: #1202992
      - CVE-2013-4127

  Also seems to be giving the fedora folks fits as well:
  https://bugzilla.redhat.com/show_bug.cgi?id=975065

  Aug 24 20:00:55 gwbvm4 kernel: [277318.536525] BUG: unable to handle kernel NULL pointer dereference at 00000000000001ea
  Aug 24 20:00:55 gwbvm4 kernel: [277318.537027] IP: [<ffffffff8113c1a5>] put_page+0x5/0x40
  Aug 24 20:00:55 gwbvm4 kernel: [277318.537359] PGD 0
  Aug 24 20:00:55 gwbvm4 kernel: [277318.537505] Oops: 0000 [#1] SMP
  Aug 24 20:00:55 gwbvm4 kernel: [277318.537716] Modules linked in: xt_recent(F) nfnetlink_log(F) nfnetlink(F) vhost_net macvtap(F) macvlan(F) brcompat(OF) openvswitch(OF) mptctl(F) mptbase(F) ipmi_devintf ipmi_si ipmi_msghandler ebtable_nat(F) ebtables(F) ipt_MASQUERADE(F) iptable_nat(F) nf_nat_ipv4(F) xt_CHECKSUM(F) iptable_mangle(F) ib_iser rdma_cm ib_addr iw_cm ib_cm ib_sa ib_mad ib_core iscsi_tcp(F) libiscsi_tcp(F) libiscsi(F) scsi_transport_iscsi(F) stp(F) llc(F) ip6t_REJECT(F) xt_hl(F) ip6t_rt(F) nf_conntrack_ipv6(F) nf_defrag_ipv6(F) ipt_REJECT(F) xt_comment(F) xt_limit(F) xt_tcpudp(F) vesafb(F) xt_addrtype(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) xt_state(F) ip6table_filter(F) ip6_tables(F) nf_conntrack_netbios_ns(F) nf_conntrack_broadcast(F) nf_nat_ftp(F) nf_nat(F) nf_conntrack_ftp(F) nf_conntrack(F) iptable_filter(F) ip_tables(F) coretemp x_tables(F) kvm_intel kvm ghash_clmulni_intel(F) aesni_intel(F) aes_x86_64(F) xts(F) lrw(F) gf128mul(F) ablk_helper(F) cryptd(F) gpio_ich lpc_ich microcode(F) serio_r
  Aug 24 20:00:55 gwbvm4 kernel: aw(F) i7core_edac mac_hid edac_core lp(F) parport(F) btrfs(F) zlib_deflate(F) libcrc32c(F) ahci(F) libahci(F) igb cxgb3 dca ptp hpsa mdio pps_core [last unloaded: bridge]
  Aug 24 20:00:55 gwbvm4 kernel: [277318.544745] CPU 0
  Aug 24 20:00:55 gwbvm4 kernel: [277318.544866] Pid: 5489, comm: vhost-5488 Tainted: GF         IO 3.8.0-29-generic #42-Ubuntu HP ProLiant DL160 G6
  Aug 24 20:00:55 gwbvm4 kernel: [277318.545560] RIP: 0010:[<ffffffff8113c1a5>]  [<ffffffff8113c1a5>] put_page+0x5/0x40
  Aug 24 20:00:55 gwbvm4 kernel: [277318.546034] RSP: 0018:ffff8817ccbc1c78  EFLAGS: 00010202
  Aug 24 20:00:55 gwbvm4 kernel: [277318.546356] RAX: ffff8809728a1ac0 RBX: 0000000000000012 RCX: ffff8809728a1ac0
  Aug 24 20:00:55 gwbvm4 kernel: [277318.569029] RDX: 0000000000000140 RSI: ffff8809728a1ac0 RDI: 00000000000001ea
  Aug 24 20:00:55 gwbvm4 kernel: [277318.592195] RBP: ffff8817ccbc1c90 R08: ffff880970704518 R09: 0000000000000010
  Aug 24 20:00:55 gwbvm4 kernel: [277318.615537] R10: 0000000000000001 R11: 0000000000000007 R12: ffff881645ca5100
  Aug 24 20:00:55 gwbvm4 kernel: [277318.639881] R13: ffffffff814dfa35 R14: 000000000000000c R15: ffff881645ca5100
  Aug 24 20:00:55 gwbvm4 kernel: [277318.664386] FS:  0000000000000000(0000) GS:ffff880c0fc00000(0000) knlGS:0000000000000000
  Aug 24 20:00:55 gwbvm4 kernel: [277318.689030] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
  Aug 24 20:00:55 gwbvm4 kernel: [277318.701226] CR2: 00000000000001ea CR3: 000000069ca0b000 CR4: 00000000000027e0
  Aug 24 20:00:55 gwbvm4 kernel: [277318.725240] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  Aug 24 20:00:55 gwbvm4 kernel: [277318.749120] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
  Aug 24 20:00:55 gwbvm4 kernel: [277318.773018] Process vhost-5488 (pid: 5489, threadinfo ffff8817ccbc0000, task ffff8817f6a80000)
  Aug 24 20:00:55 gwbvm4 kernel: [277318.796899] Stack:
  Aug 24 20:00:55 gwbvm4 kernel: [277318.808406]  ffffffff815bcebf ffff881645ca5100 ffff881645ca5100 ffff8817ccbc1ca8
  Aug 24 20:00:55 gwbvm4 kernel: [277318.831675]  ffffffff815bcf5a ffff8809707043d8 ffff8817ccbc1cd0 ffffffff815bd012
  Aug 24 20:00:55 gwbvm4 kernel: [277318.855551]  ffff8809707043d8 000000000000f4ee ffff880a2c048800 ffff8817ccbc1d58
  Aug 24 20:00:55 gwbvm4 kernel: [277318.879408] Call Trace:
  Aug 24 20:00:55 gwbvm4 kernel: [277318.891138]  [<ffffffff815bcebf>] ? skb_release_data+0x8f/0x110
  Aug 24 20:00:55 gwbvm4 kernel: [277318.903096]  [<ffffffff815bcf5a>] __kfree_skb+0x1a/0xa0
  Aug 24 20:00:55 gwbvm4 kernel: [277318.914622]  [<ffffffff815bd012>] kfree_skb+0x32/0x90
  Aug 24 20:00:55 gwbvm4 kernel: [277318.925947]  [<ffffffff814dfa35>] tun_get_user+0x5f5/0x720
  Aug 24 20:00:55 gwbvm4 kernel: [277318.937089]  [<ffffffff814dfbb7>] tun_sendmsg+0x57/0x80
  Aug 24 20:00:55 gwbvm4 kernel: [277318.947987]  [<ffffffffa0435656>] handle_tx+0x266/0x580 [vhost_net]
  Aug 24 20:00:55 gwbvm4 kernel: [277318.958700]  [<ffffffffa04359a5>] handle_tx_kick+0x15/0x20 [vhost_net]
  Aug 24 20:00:55 gwbvm4 kernel: [277318.969222]  [<ffffffffa043295f>] vhost_worker+0xff/0x1b0 [vhost_net]
  Aug 24 20:00:55 gwbvm4 kernel: [277318.979546]  [<ffffffffa0432860>] ? vhost_work_flush+0x130/0x130 [vhost_net]
  Aug 24 20:00:55 gwbvm4 kernel: [277318.989849]  [<ffffffff8107d590>] kthread+0xc0/0xd0
  Aug 24 20:00:55 gwbvm4 kernel: [277319.000170]  [<ffffffff8107d4d0>] ? kthread_create_on_node+0x120/0x120
  Aug 24 20:00:55 gwbvm4 kernel: [277319.010475]  [<ffffffff816d556c>] ret_from_fork+0x7c/0xb0
  Aug 24 20:00:55 gwbvm4 kernel: [277319.020339]  [<ffffffff8107d4d0>] ? kthread_create_on_node+0x120/0x120
  Aug 24 20:00:55 gwbvm4 kernel: [277319.030361] Code: fc 00 00 00 00 e8 ac fe ff ff 48 63 45 fc 65 48 01 04 25 78 08 01 00 c9 c3 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 <48> f7 07 00 c0 00 00 55 48 89 e5 75 15 f0 ff 4f 1c 0f 94 c0 84
  Aug 24 20:00:55 gwbvm4 kernel: [277319.060251] RIP  [<ffffffff8113c1a5>] put_page+0x5/0x40
  Aug 24 20:00:55 gwbvm4 kernel: [277319.069747]  RSP <ffff8817ccbc1c78>
  Aug 24 20:00:55 gwbvm4 kernel: [277319.078868] CR2: 00000000000001ea
  Aug 24 20:00:55 gwbvm4 kernel: [277319.102160] ---[ end trace def21f8b2fed77aa ]---

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1216444/+subscriptions