kernel-packages team mailing list archive
-
kernel-packages team
-
Mailing list archive
-
Message #92664
[Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
HVM instance would have the same issue when using PV network drivers
(which usually they do for performance). However one also needs to cause
fragmented skbs which contain multiple compound page fragments. And that
depends on many factors which may not always be easy to meet.
By now, there actually seems to be a work-around that has been applied
to upstream v3.17. Looks like we have to pick the following (or actually
get it into the stable process):
commit 97a6d1bb2b658ac85ed88205ccd1ab809899884d
Author: Zoltan Kiss <zoltan.kiss@xxxxxxxxxx>
Date: Mon Aug 11 18:32:23 2014 +0100
xen-netfront: Fix handling packets on compound pages with
skb_linearize
** Also affects: linux (Ubuntu Utopic)
Importance: Undecided
Status: New
** Also affects: linux (Ubuntu Trusty)
Importance: Undecided
Status: New
** Changed in: linux (Ubuntu Trusty)
Importance: Undecided => Medium
** Changed in: linux (Ubuntu Trusty)
Status: New => Triaged
** Changed in: linux (Ubuntu Utopic)
Importance: Undecided => Medium
** Changed in: linux (Ubuntu Utopic)
Status: New => Triaged
** Description changed:
Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the
kernel log:
- xen_netfront: xennet: skb rides the rocket: 19 slots
+ xen_netfront: xennet: skb rides the rocket: 19 slots
Each of these messages corresponds to a dropped TX packet, and
eventually causes our application's connections to break and timeout.
The problem appears when network load increases. We have Node.js
processes doing pubsub with a Redis server, and these are most visibly
affected, showing frequent connection loss. The processes talk to each
other using the private addresses EC2 allocates to the machines.
Notably, the default MTU on the network interface seems to have gone up
from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500
seems to drastically reduce dropped packets. (Can't say for certain if
it completely eliminates the problem.)
The machines we run are started from ami-896c96fe.
ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: linux-image-3.13.0-24-generic 3.13.0-24.46
ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9
Uname: Linux 3.13.0-24-generic x86_64
AlsaDevices:
- total 0
- crw-rw---- 1 root audio 116, 1 May 9 09:01 seq
- crw-rw---- 1 root audio 116, 33 May 9 09:01 timer
+ total 0
+ crw-rw---- 1 root audio 116, 1 May 9 09:01 seq
+ crw-rw---- 1 root audio 116, 33 May 9 09:01 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.14.1-0ubuntu3
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory: 'iw'
Date: Fri May 9 09:11:18 2014
Ec2AMI: ami-896c96fe
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: eu-west-1c
Ec2InstanceType: c3.large
Ec2Kernel: aki-52a34525
Ec2Ramdisk: unavailable
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lspci:
-
+
Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
PciMultimedia:
-
+
ProcFB:
-
+
ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0
RelatedPackageVersions:
- linux-restricted-modules-3.13.0-24-generic N/A
- linux-backports-modules-3.13.0-24-generic N/A
- linux-firmware N/A
+ linux-restricted-modules-3.13.0-24-generic N/A
+ linux-backports-modules-3.13.0-24-generic N/A
+ linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
- ---
+ ---
AlsaDevices:
- total 0
- crw-rw---- 1 root audio 116, 1 May 9 09:54 seq
- crw-rw---- 1 root audio 116, 33 May 9 09:54 timer
+ total 0
+ crw-rw---- 1 root audio 116, 1 May 9 09:54 seq
+ crw-rw---- 1 root audio 116, 33 May 9 09:54 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning
DistroRelease: Ubuntu 14.04
Ec2AMI: ami-896c96fe
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: eu-west-1c
Ec2InstanceType: c3.large
Ec2Kernel: aki-52a34525
Ec2Ramdisk: unavailable
IwConfig: Error: [Errno 2] No such file or directory
Lspci:
-
+
Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
Package: linux (not installed)
PciMultimedia:
-
+
ProcFB:
-
+
ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0
ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9
RelatedPackageVersions:
- linux-restricted-modules-3.13.0-24-generic N/A
- linux-backports-modules-3.13.0-24-generic N/A
- linux-firmware N/A
+ linux-restricted-modules-3.13.0-24-generic N/A
+ linux-backports-modules-3.13.0-24-generic N/A
+ linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory
Tags: trusty ec2-images
Uname: Linux 3.13.0-24-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video
_MarkForUpload: True
+
+ break-fix: - 97a6d1bb2b658ac85ed88205ccd1ab809899884d
** Tags added: kernel-bug-break-fix
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1317811
Title:
Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x
slots"
Status in linux package in Ubuntu:
In Progress
Status in linux source package in Trusty:
Triaged
Status in linux source package in Utopic:
Triaged
Bug description:
Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the
kernel log:
xen_netfront: xennet: skb rides the rocket: 19 slots
Each of these messages corresponds to a dropped TX packet, and
eventually causes our application's connections to break and timeout.
The problem appears when network load increases. We have Node.js
processes doing pubsub with a Redis server, and these are most visibly
affected, showing frequent connection loss. The processes talk to each
other using the private addresses EC2 allocates to the machines.
Notably, the default MTU on the network interface seems to have gone
up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to
1500 seems to drastically reduce dropped packets. (Can't say for
certain if it completely eliminates the problem.)
The machines we run are started from ami-896c96fe.
ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: linux-image-3.13.0-24-generic 3.13.0-24.46
ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9
Uname: Linux 3.13.0-24-generic x86_64
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 May 9 09:01 seq
crw-rw---- 1 root audio 116, 33 May 9 09:01 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.14.1-0ubuntu3
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory: 'iw'
Date: Fri May 9 09:11:18 2014
Ec2AMI: ami-896c96fe
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: eu-west-1c
Ec2InstanceType: c3.large
Ec2Kernel: aki-52a34525
Ec2Ramdisk: unavailable
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lspci:
Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
PciMultimedia:
ProcFB:
ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0
RelatedPackageVersions:
linux-restricted-modules-3.13.0-24-generic N/A
linux-backports-modules-3.13.0-24-generic N/A
linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
---
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 May 9 09:54 seq
crw-rw---- 1 root audio 116, 33 May 9 09:54 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning
DistroRelease: Ubuntu 14.04
Ec2AMI: ami-896c96fe
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: eu-west-1c
Ec2InstanceType: c3.large
Ec2Kernel: aki-52a34525
Ec2Ramdisk: unavailable
IwConfig: Error: [Errno 2] No such file or directory
Lspci:
Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
Package: linux (not installed)
PciMultimedia:
ProcFB:
ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0
ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9
RelatedPackageVersions:
linux-restricted-modules-3.13.0-24-generic N/A
linux-backports-modules-3.13.0-24-generic N/A
linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory
Tags: trusty ec2-images
Uname: Linux 3.13.0-24-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video
_MarkForUpload: True
break-fix: - 97a6d1bb2b658ac85ed88205ccd1ab809899884d
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions
References
