← Back to team overview

kernel-packages team mailing list archive

  1. kernel-packages team
  2. Mailing list archive
  3. Message #09671

[Bug 1216444] Re: Vhost-net made unstable by linux_3.8.0-28.41

  Thread Previous • Date Previous • • Thread Next 
apport information

** Tags added: apport-collected

** Description changed:

  On multiple machines with vm's using vhost-net this bug takes out the
  guest network interface under load.  The vm is only able to see
  broadcast traffic after this happens.  Started happening immediately
  after upgrading from linux-image-3.8.0-27-generic to linux-
  image-3.8.0-29-generic.  This changelog entry from linux_3.8.0-28.41
  seems particularly relevant:
  
    * vhost-net: fix use-after-free in vhost_net_flush
      - LP: #1202992
      - CVE-2013-4127
  
  Also seems to be giving the fedora folks fits as well:
  https://bugzilla.redhat.com/show_bug.cgi?id=975065
  
  Aug 24 20:00:55 gwbvm4 kernel: [277318.536525] BUG: unable to handle kernel NULL pointer dereference at 00000000000001ea
  Aug 24 20:00:55 gwbvm4 kernel: [277318.537027] IP: [<ffffffff8113c1a5>] put_page+0x5/0x40
  Aug 24 20:00:55 gwbvm4 kernel: [277318.537359] PGD 0
  Aug 24 20:00:55 gwbvm4 kernel: [277318.537505] Oops: 0000 [#1] SMP
  Aug 24 20:00:55 gwbvm4 kernel: [277318.537716] Modules linked in: xt_recent(F) nfnetlink_log(F) nfnetlink(F) vhost_net macvtap(F) macvlan(F) brcompat(OF) openvswitch(OF) mptctl(F) mptbase(F) ipmi_devintf ipmi_si ipmi_msghandler ebtable_nat(F) ebtables(F) ipt_MASQUERADE(F) iptable_nat(F) nf_nat_ipv4(F) xt_CHECKSUM(F) iptable_mangle(F) ib_iser rdma_cm ib_addr iw_cm ib_cm ib_sa ib_mad ib_core iscsi_tcp(F) libiscsi_tcp(F) libiscsi(F) scsi_transport_iscsi(F) stp(F) llc(F) ip6t_REJECT(F) xt_hl(F) ip6t_rt(F) nf_conntrack_ipv6(F) nf_defrag_ipv6(F) ipt_REJECT(F) xt_comment(F) xt_limit(F) xt_tcpudp(F) vesafb(F) xt_addrtype(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) xt_state(F) ip6table_filter(F) ip6_tables(F) nf_conntrack_netbios_ns(F) nf_conntrack_broadcast(F) nf_nat_ftp(F) nf_nat(F) nf_conntrack_ftp(F) nf_conntrack(F) iptable_filter(F) ip_tables(F) coretemp x_tables(F) kvm_intel kvm ghash_clmulni_intel(F) aesni_intel(F) aes_x86_64(F) xts(F) lrw(F) gf128mul(F) ablk_helper(F) cryptd(F) gpio_ich lpc_ich microcode(F) serio_r
  Aug 24 20:00:55 gwbvm4 kernel: aw(F) i7core_edac mac_hid edac_core lp(F) parport(F) btrfs(F) zlib_deflate(F) libcrc32c(F) ahci(F) libahci(F) igb cxgb3 dca ptp hpsa mdio pps_core [last unloaded: bridge]
  Aug 24 20:00:55 gwbvm4 kernel: [277318.544745] CPU 0
  Aug 24 20:00:55 gwbvm4 kernel: [277318.544866] Pid: 5489, comm: vhost-5488 Tainted: GF         IO 3.8.0-29-generic #42-Ubuntu HP ProLiant DL160 G6
  Aug 24 20:00:55 gwbvm4 kernel: [277318.545560] RIP: 0010:[<ffffffff8113c1a5>]  [<ffffffff8113c1a5>] put_page+0x5/0x40
  Aug 24 20:00:55 gwbvm4 kernel: [277318.546034] RSP: 0018:ffff8817ccbc1c78  EFLAGS: 00010202
  Aug 24 20:00:55 gwbvm4 kernel: [277318.546356] RAX: ffff8809728a1ac0 RBX: 0000000000000012 RCX: ffff8809728a1ac0
  Aug 24 20:00:55 gwbvm4 kernel: [277318.569029] RDX: 0000000000000140 RSI: ffff8809728a1ac0 RDI: 00000000000001ea
  Aug 24 20:00:55 gwbvm4 kernel: [277318.592195] RBP: ffff8817ccbc1c90 R08: ffff880970704518 R09: 0000000000000010
  Aug 24 20:00:55 gwbvm4 kernel: [277318.615537] R10: 0000000000000001 R11: 0000000000000007 R12: ffff881645ca5100
  Aug 24 20:00:55 gwbvm4 kernel: [277318.639881] R13: ffffffff814dfa35 R14: 000000000000000c R15: ffff881645ca5100
  Aug 24 20:00:55 gwbvm4 kernel: [277318.664386] FS:  0000000000000000(0000) GS:ffff880c0fc00000(0000) knlGS:0000000000000000
  Aug 24 20:00:55 gwbvm4 kernel: [277318.689030] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
  Aug 24 20:00:55 gwbvm4 kernel: [277318.701226] CR2: 00000000000001ea CR3: 000000069ca0b000 CR4: 00000000000027e0
  Aug 24 20:00:55 gwbvm4 kernel: [277318.725240] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  Aug 24 20:00:55 gwbvm4 kernel: [277318.749120] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
  Aug 24 20:00:55 gwbvm4 kernel: [277318.773018] Process vhost-5488 (pid: 5489, threadinfo ffff8817ccbc0000, task ffff8817f6a80000)
  Aug 24 20:00:55 gwbvm4 kernel: [277318.796899] Stack:
  Aug 24 20:00:55 gwbvm4 kernel: [277318.808406]  ffffffff815bcebf ffff881645ca5100 ffff881645ca5100 ffff8817ccbc1ca8
  Aug 24 20:00:55 gwbvm4 kernel: [277318.831675]  ffffffff815bcf5a ffff8809707043d8 ffff8817ccbc1cd0 ffffffff815bd012
  Aug 24 20:00:55 gwbvm4 kernel: [277318.855551]  ffff8809707043d8 000000000000f4ee ffff880a2c048800 ffff8817ccbc1d58
  Aug 24 20:00:55 gwbvm4 kernel: [277318.879408] Call Trace:
  Aug 24 20:00:55 gwbvm4 kernel: [277318.891138]  [<ffffffff815bcebf>] ? skb_release_data+0x8f/0x110
  Aug 24 20:00:55 gwbvm4 kernel: [277318.903096]  [<ffffffff815bcf5a>] __kfree_skb+0x1a/0xa0
  Aug 24 20:00:55 gwbvm4 kernel: [277318.914622]  [<ffffffff815bd012>] kfree_skb+0x32/0x90
  Aug 24 20:00:55 gwbvm4 kernel: [277318.925947]  [<ffffffff814dfa35>] tun_get_user+0x5f5/0x720
  Aug 24 20:00:55 gwbvm4 kernel: [277318.937089]  [<ffffffff814dfbb7>] tun_sendmsg+0x57/0x80
  Aug 24 20:00:55 gwbvm4 kernel: [277318.947987]  [<ffffffffa0435656>] handle_tx+0x266/0x580 [vhost_net]
  Aug 24 20:00:55 gwbvm4 kernel: [277318.958700]  [<ffffffffa04359a5>] handle_tx_kick+0x15/0x20 [vhost_net]
  Aug 24 20:00:55 gwbvm4 kernel: [277318.969222]  [<ffffffffa043295f>] vhost_worker+0xff/0x1b0 [vhost_net]
  Aug 24 20:00:55 gwbvm4 kernel: [277318.979546]  [<ffffffffa0432860>] ? vhost_work_flush+0x130/0x130 [vhost_net]
  Aug 24 20:00:55 gwbvm4 kernel: [277318.989849]  [<ffffffff8107d590>] kthread+0xc0/0xd0
  Aug 24 20:00:55 gwbvm4 kernel: [277319.000170]  [<ffffffff8107d4d0>] ? kthread_create_on_node+0x120/0x120
  Aug 24 20:00:55 gwbvm4 kernel: [277319.010475]  [<ffffffff816d556c>] ret_from_fork+0x7c/0xb0
  Aug 24 20:00:55 gwbvm4 kernel: [277319.020339]  [<ffffffff8107d4d0>] ? kthread_create_on_node+0x120/0x120
  Aug 24 20:00:55 gwbvm4 kernel: [277319.030361] Code: fc 00 00 00 00 e8 ac fe ff ff 48 63 45 fc 65 48 01 04 25 78 08 01 00 c9 c3 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 <48> f7 07 00 c0 00 00 55 48 89 e5 75 15 f0 ff 4f 1c 0f 94 c0 84
  Aug 24 20:00:55 gwbvm4 kernel: [277319.060251] RIP  [<ffffffff8113c1a5>] put_page+0x5/0x40
  Aug 24 20:00:55 gwbvm4 kernel: [277319.069747]  RSP <ffff8817ccbc1c78>
  Aug 24 20:00:55 gwbvm4 kernel: [277319.078868] CR2: 00000000000001ea
  Aug 24 20:00:55 gwbvm4 kernel: [277319.102160] ---[ end trace def21f8b2fed77aa ]---
+ --- 
+ AlsaDevices:
+  total 0
+  crw-rw---T 1 root audio 116,  1 Aug 24 20:18 seq
+  crw-rw---T 1 root audio 116, 33 Aug 24 20:18 timer
+ AplayDevices: Error: [Errno 2] No such file or directory
+ ApportVersion: 2.9.2-0ubuntu8.3
+ Architecture: amd64
+ ArecordDevices: Error: [Errno 2] No such file or directory
+ AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
+ CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
+ DistroRelease: Ubuntu 13.04
+ InstallationDate: Installed on 2012-01-23 (579 days ago)
+ InstallationMedia: Ubuntu-Server 11.10 "Oneiric Ocelot" - Release amd64 (20111011)
+ MachineType: HP ProLiant DL160 G6
+ MarkForUpload: True
+ Package: linux (not installed)
+ PciMultimedia:
+  
+ ProcEnviron:
+  TERM=xterm
+  PATH=(custom, no user)
+  LANG=en_US.UTF-8
+  SHELL=/bin/bash
+ ProcFB: 0 VESA VGA
+ ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.8.0-29-generic root=UUID=39c8e6cb-2cfa-41e5-a387-ab847da8a3a7 ro
+ ProcVersionSignature: Ubuntu 3.8.0-29.42-generic 3.8.13.5
+ RelatedPackageVersions:
+  linux-restricted-modules-3.8.0-29-generic N/A
+  linux-backports-modules-3.8.0-29-generic  N/A
+  linux-firmware                            1.106
+ RfKill: Error: [Errno 2] No such file or directory
+ Tags:  raring
+ Uname: Linux 3.8.0-29-generic x86_64
+ UpgradeStatus: Upgraded to raring on 2013-05-07 (110 days ago)
+ UserGroups:
+  
+ WifiSyslog: Aug 25 13:47:27 gwbvm4 kernel: [62858.061374] sda1: WRITE SAME failed. Manually zeroing.
+ dmi.bios.date: 07/06/2011
+ dmi.bios.vendor: HP
+ dmi.bios.version: O33
+ dmi.chassis.type: 23
+ dmi.chassis.vendor: HP
+ dmi.modalias: dmi:bvnHP:bvrO33:bd07/06/2011:svnHP:pnProLiantDL160G6:pvr:cvnHP:ct23:cvr:
+ dmi.product.name: ProLiant DL160 G6
+ dmi.sys.vendor: HP

** Attachment added: "BootDmesg.txt"
   https://bugs.launchpad.net/bugs/1216444/+attachment/3787255/+files/BootDmesg.txt

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1216444

Title:
  Vhost-net made unstable by linux_3.8.0-28.41

Status in “linux” package in Ubuntu:
  Confirmed

Bug description:
  On multiple machines with vm's using vhost-net this bug takes out the
  guest network interface under load.  The vm is only able to see
  broadcast traffic after this happens.  Started happening immediately
  after upgrading from linux-image-3.8.0-27-generic to linux-
  image-3.8.0-29-generic.  This changelog entry from linux_3.8.0-28.41
  seems particularly relevant:

    * vhost-net: fix use-after-free in vhost_net_flush
      - LP: #1202992
      - CVE-2013-4127

  Also seems to be giving the fedora folks fits as well:
  https://bugzilla.redhat.com/show_bug.cgi?id=975065

  Aug 24 20:00:55 gwbvm4 kernel: [277318.536525] BUG: unable to handle kernel NULL pointer dereference at 00000000000001ea
  Aug 24 20:00:55 gwbvm4 kernel: [277318.537027] IP: [<ffffffff8113c1a5>] put_page+0x5/0x40
  Aug 24 20:00:55 gwbvm4 kernel: [277318.537359] PGD 0
  Aug 24 20:00:55 gwbvm4 kernel: [277318.537505] Oops: 0000 [#1] SMP
  Aug 24 20:00:55 gwbvm4 kernel: [277318.537716] Modules linked in: xt_recent(F) nfnetlink_log(F) nfnetlink(F) vhost_net macvtap(F) macvlan(F) brcompat(OF) openvswitch(OF) mptctl(F) mptbase(F) ipmi_devintf ipmi_si ipmi_msghandler ebtable_nat(F) ebtables(F) ipt_MASQUERADE(F) iptable_nat(F) nf_nat_ipv4(F) xt_CHECKSUM(F) iptable_mangle(F) ib_iser rdma_cm ib_addr iw_cm ib_cm ib_sa ib_mad ib_core iscsi_tcp(F) libiscsi_tcp(F) libiscsi(F) scsi_transport_iscsi(F) stp(F) llc(F) ip6t_REJECT(F) xt_hl(F) ip6t_rt(F) nf_conntrack_ipv6(F) nf_defrag_ipv6(F) ipt_REJECT(F) xt_comment(F) xt_limit(F) xt_tcpudp(F) vesafb(F) xt_addrtype(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) xt_state(F) ip6table_filter(F) ip6_tables(F) nf_conntrack_netbios_ns(F) nf_conntrack_broadcast(F) nf_nat_ftp(F) nf_nat(F) nf_conntrack_ftp(F) nf_conntrack(F) iptable_filter(F) ip_tables(F) coretemp x_tables(F) kvm_intel kvm ghash_clmulni_intel(F) aesni_intel(F) aes_x86_64(F) xts(F) lrw(F) gf128mul(F) ablk_helper(F) cryptd(F) gpio_ich lpc_ich microcode(F) serio_r
  Aug 24 20:00:55 gwbvm4 kernel: aw(F) i7core_edac mac_hid edac_core lp(F) parport(F) btrfs(F) zlib_deflate(F) libcrc32c(F) ahci(F) libahci(F) igb cxgb3 dca ptp hpsa mdio pps_core [last unloaded: bridge]
  Aug 24 20:00:55 gwbvm4 kernel: [277318.544745] CPU 0
  Aug 24 20:00:55 gwbvm4 kernel: [277318.544866] Pid: 5489, comm: vhost-5488 Tainted: GF         IO 3.8.0-29-generic #42-Ubuntu HP ProLiant DL160 G6
  Aug 24 20:00:55 gwbvm4 kernel: [277318.545560] RIP: 0010:[<ffffffff8113c1a5>]  [<ffffffff8113c1a5>] put_page+0x5/0x40
  Aug 24 20:00:55 gwbvm4 kernel: [277318.546034] RSP: 0018:ffff8817ccbc1c78  EFLAGS: 00010202
  Aug 24 20:00:55 gwbvm4 kernel: [277318.546356] RAX: ffff8809728a1ac0 RBX: 0000000000000012 RCX: ffff8809728a1ac0
  Aug 24 20:00:55 gwbvm4 kernel: [277318.569029] RDX: 0000000000000140 RSI: ffff8809728a1ac0 RDI: 00000000000001ea
  Aug 24 20:00:55 gwbvm4 kernel: [277318.592195] RBP: ffff8817ccbc1c90 R08: ffff880970704518 R09: 0000000000000010
  Aug 24 20:00:55 gwbvm4 kernel: [277318.615537] R10: 0000000000000001 R11: 0000000000000007 R12: ffff881645ca5100
  Aug 24 20:00:55 gwbvm4 kernel: [277318.639881] R13: ffffffff814dfa35 R14: 000000000000000c R15: ffff881645ca5100
  Aug 24 20:00:55 gwbvm4 kernel: [277318.664386] FS:  0000000000000000(0000) GS:ffff880c0fc00000(0000) knlGS:0000000000000000
  Aug 24 20:00:55 gwbvm4 kernel: [277318.689030] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
  Aug 24 20:00:55 gwbvm4 kernel: [277318.701226] CR2: 00000000000001ea CR3: 000000069ca0b000 CR4: 00000000000027e0
  Aug 24 20:00:55 gwbvm4 kernel: [277318.725240] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  Aug 24 20:00:55 gwbvm4 kernel: [277318.749120] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
  Aug 24 20:00:55 gwbvm4 kernel: [277318.773018] Process vhost-5488 (pid: 5489, threadinfo ffff8817ccbc0000, task ffff8817f6a80000)
  Aug 24 20:00:55 gwbvm4 kernel: [277318.796899] Stack:
  Aug 24 20:00:55 gwbvm4 kernel: [277318.808406]  ffffffff815bcebf ffff881645ca5100 ffff881645ca5100 ffff8817ccbc1ca8
  Aug 24 20:00:55 gwbvm4 kernel: [277318.831675]  ffffffff815bcf5a ffff8809707043d8 ffff8817ccbc1cd0 ffffffff815bd012
  Aug 24 20:00:55 gwbvm4 kernel: [277318.855551]  ffff8809707043d8 000000000000f4ee ffff880a2c048800 ffff8817ccbc1d58
  Aug 24 20:00:55 gwbvm4 kernel: [277318.879408] Call Trace:
  Aug 24 20:00:55 gwbvm4 kernel: [277318.891138]  [<ffffffff815bcebf>] ? skb_release_data+0x8f/0x110
  Aug 24 20:00:55 gwbvm4 kernel: [277318.903096]  [<ffffffff815bcf5a>] __kfree_skb+0x1a/0xa0
  Aug 24 20:00:55 gwbvm4 kernel: [277318.914622]  [<ffffffff815bd012>] kfree_skb+0x32/0x90
  Aug 24 20:00:55 gwbvm4 kernel: [277318.925947]  [<ffffffff814dfa35>] tun_get_user+0x5f5/0x720
  Aug 24 20:00:55 gwbvm4 kernel: [277318.937089]  [<ffffffff814dfbb7>] tun_sendmsg+0x57/0x80
  Aug 24 20:00:55 gwbvm4 kernel: [277318.947987]  [<ffffffffa0435656>] handle_tx+0x266/0x580 [vhost_net]
  Aug 24 20:00:55 gwbvm4 kernel: [277318.958700]  [<ffffffffa04359a5>] handle_tx_kick+0x15/0x20 [vhost_net]
  Aug 24 20:00:55 gwbvm4 kernel: [277318.969222]  [<ffffffffa043295f>] vhost_worker+0xff/0x1b0 [vhost_net]
  Aug 24 20:00:55 gwbvm4 kernel: [277318.979546]  [<ffffffffa0432860>] ? vhost_work_flush+0x130/0x130 [vhost_net]
  Aug 24 20:00:55 gwbvm4 kernel: [277318.989849]  [<ffffffff8107d590>] kthread+0xc0/0xd0
  Aug 24 20:00:55 gwbvm4 kernel: [277319.000170]  [<ffffffff8107d4d0>] ? kthread_create_on_node+0x120/0x120
  Aug 24 20:00:55 gwbvm4 kernel: [277319.010475]  [<ffffffff816d556c>] ret_from_fork+0x7c/0xb0
  Aug 24 20:00:55 gwbvm4 kernel: [277319.020339]  [<ffffffff8107d4d0>] ? kthread_create_on_node+0x120/0x120
  Aug 24 20:00:55 gwbvm4 kernel: [277319.030361] Code: fc 00 00 00 00 e8 ac fe ff ff 48 63 45 fc 65 48 01 04 25 78 08 01 00 c9 c3 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 <48> f7 07 00 c0 00 00 55 48 89 e5 75 15 f0 ff 4f 1c 0f 94 c0 84
  Aug 24 20:00:55 gwbvm4 kernel: [277319.060251] RIP  [<ffffffff8113c1a5>] put_page+0x5/0x40
  Aug 24 20:00:55 gwbvm4 kernel: [277319.069747]  RSP <ffff8817ccbc1c78>
  Aug 24 20:00:55 gwbvm4 kernel: [277319.078868] CR2: 00000000000001ea
  Aug 24 20:00:55 gwbvm4 kernel: [277319.102160] ---[ end trace def21f8b2fed77aa ]---
  --- 
  AlsaDevices:
   total 0
   crw-rw---T 1 root audio 116,  1 Aug 24 20:18 seq
   crw-rw---T 1 root audio 116, 33 Aug 24 20:18 timer
  AplayDevices: Error: [Errno 2] No such file or directory
  ApportVersion: 2.9.2-0ubuntu8.3
  Architecture: amd64
  ArecordDevices: Error: [Errno 2] No such file or directory
  AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
  CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
  DistroRelease: Ubuntu 13.04
  InstallationDate: Installed on 2012-01-23 (579 days ago)
  InstallationMedia: Ubuntu-Server 11.10 "Oneiric Ocelot" - Release amd64 (20111011)
  MachineType: HP ProLiant DL160 G6
  MarkForUpload: True
  Package: linux (not installed)
  PciMultimedia:
   
  ProcEnviron:
   TERM=xterm
   PATH=(custom, no user)
   LANG=en_US.UTF-8
   SHELL=/bin/bash
  ProcFB: 0 VESA VGA
  ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.8.0-29-generic root=UUID=39c8e6cb-2cfa-41e5-a387-ab847da8a3a7 ro
  ProcVersionSignature: Ubuntu 3.8.0-29.42-generic 3.8.13.5
  RelatedPackageVersions:
   linux-restricted-modules-3.8.0-29-generic N/A
   linux-backports-modules-3.8.0-29-generic  N/A
   linux-firmware                            1.106
  RfKill: Error: [Errno 2] No such file or directory
  Tags:  raring
  Uname: Linux 3.8.0-29-generic x86_64
  UpgradeStatus: Upgraded to raring on 2013-05-07 (110 days ago)
  UserGroups:
   
  WifiSyslog: Aug 25 13:47:27 gwbvm4 kernel: [62858.061374] sda1: WRITE SAME failed. Manually zeroing.
  dmi.bios.date: 07/06/2011
  dmi.bios.vendor: HP
  dmi.bios.version: O33
  dmi.chassis.type: 23
  dmi.chassis.vendor: HP
  dmi.modalias: dmi:bvnHP:bvrO33:bd07/06/2011:svnHP:pnProLiantDL160G6:pvr:cvnHP:ct23:cvr:
  dmi.product.name: ProLiant DL160 G6
  dmi.sys.vendor: HP

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1216444/+subscriptions

References

  Thread Previous • Date Previous • • Thread Next