kernel-packages team mailing list archive

[Rodrigo Vaz <1391339@xxxxxxxxxxxxxxxxxx>]

Hi Stefan,

Ok I think I've figured out why you were unable to reproduce the
slowness, as I mentioned earlier we use the m2 instance type that runs
on underlying Xen 3.4 where the t1.micro is probably running on newer
infrastructure so I decided to give m3 a try and xen is actually newer
(4.2) following is the result of your iperf script on m2 and m3
instances and as you can see the old instance is really affected:

m3 client / m3 server / gro on
Average bandwidth = 958.334 Mbits/sec
StdDev            = 21.3581 Mbits/sec

Server Output Sample:
[SUM]  0.0-50.5 sec  5.87 GBytes  1000 Mbits/sec
[  5] local ... port 5001 connected with ... port 36291
[  5]  0.0-10.1 sec  1.16 GBytes   984 Mbits/sec
[  4] local ... port 5001 connected with ... port 36292
[  4]  0.0-10.1 sec  1.19 GBytes  1.01 Gbits/sec
[  5] local ... port 5001 connected with ... port 36293
[  5]  0.0-10.1 sec  1.17 GBytes   997 Mbits/sec
[  6] local ... port 5001 connected with ... port 36294
[  6]  0.0-20.2 sec  1.17 GBytes   498 Mbits/sec
[SUM]  0.0-20.2 sec  2.34 GBytes   995 Mbits/sec


m2 client / m3 server / gro on
Average bandwidth = 643.833 Mbits/sec
StdDev            = 31.689 Mbits/sec

m2 client / m3 server / gro off
Average bandwidth = 643.744 Mbits/sec
StdDev            = 29.4622 Mbits/sec

Server Output Sample:
[  6]  0.0-181.1 sec   803 MBytes  37.2 Mbits/sec
[  5] local ... port 5001 connected with ... port 32660
[  4]  0.0-191.1 sec   802 MBytes  35.2 Mbits/sec
[  6] local ... port 5001 connected with ... port 32661
[  5]  0.0-201.2 sec   803 MBytes  33.5 Mbits/sec
[  4] local ... port 5001 connected with ... port 32662
[  6]  0.0-211.2 sec   782 MBytes  31.1 Mbits/sec
[  5] local ... port 5001 connected with ... port 32663
[  4]  0.0-221.3 sec   798 MBytes  30.2 Mbits/sec
[  6] local ... port 5001 connected with ... port 32664
[  5]  0.0-231.4 sec   795 MBytes  28.8 Mbits/sec
[  4] local ... port 5001 connected with ... port 32665
[  6]  0.0-241.4 sec   796 MBytes  27.7 Mbits/sec
[  4]  0.0-251.5 sec   801 MBytes  26.7 Mbits/sec

So looks like the problem happens with Xen prior to 4.2 and I still
think it is a regression since the kernel 3.11 handles inbound network
traffic better with or without gro.

Thanks,
Rodrigo.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1391339

Title:
  Trusty kernel inbound network performance regression when GRO is
  enabled

Status in “linux” package in Ubuntu:
  Confirmed

Bug description:
  After upgrading our EC2 instances from Lucid to Trusty we noticed an
  increase on download times, Lucid instances were able to download
  twice as fast as Trusty. After some investigation and testing older
  kernels (precise, raring and saucy) we confirmed that this only
  happens on trusty kernel or newer since utopic kernel shows the same
  result and disabling gro with `ethtool -K eth0 gro off` seems to fix
  the problem making download speed the same as the Lucid instances
  again.

  The problem is easily reproducible using Apache Bench a couple times
  on files bigger than 100MB on 1Gb network (EC2) using HTTP or HTTPS.

  Following is an example of download throughput with and without gro:

  root@runtime-common.22 ~# ethtool -K eth0 gro off
  root@runtime-common.22 ~# for i in {1..10}; do ab -n 10 $URL | grep "Transfer rate"; done
  Transfer rate:          85183.40 [Kbytes/sec] received
  Transfer rate:          86375.80 [Kbytes/sec] received
  Transfer rate:          94720.24 [Kbytes/sec] received
  Transfer rate:          84783.82 [Kbytes/sec] received
  Transfer rate:          84933.09 [Kbytes/sec] received
  Transfer rate:          84714.04 [Kbytes/sec] received
  Transfer rate:          84795.58 [Kbytes/sec] received
  Transfer rate:          84636.54 [Kbytes/sec] received
  Transfer rate:          84924.26 [Kbytes/sec] received
  Transfer rate:          84994.10 [Kbytes/sec] received
  root@runtime-common.22 ~# ethtool -K eth0 gro on
  root@runtime-common.22 ~# for i in {1..10}; do ab -n 10 $URL | grep "Transfer rate"; done
  Transfer rate:          74193.53 [Kbytes/sec] received
  Transfer rate:          56808.91 [Kbytes/sec] received
  Transfer rate:          56011.58 [Kbytes/sec] received
  Transfer rate:          82227.74 [Kbytes/sec] received
  Transfer rate:          70806.54 [Kbytes/sec] received
  Transfer rate:          72848.10 [Kbytes/sec] received
  Transfer rate:          58451.94 [Kbytes/sec] received
  Transfer rate:          61221.33 [Kbytes/sec] received
  Transfer rate:          58620.21 [Kbytes/sec] received
  Transfer rate:          69950.03 [Kbytes/sec] received
  root@runtime-common.22 ~# 

  Similar results can be observed using iperf and netperf as well.

  Tested kernels: 
  Not affected: 3.8.0-44-generic (precise/raring), 3.11.0-26-generic (saucy)
  Affected: 3.13.0-39-generic (trusty), 3.16.0-24-generic (utopic)

  Let me know if I can provide any other information that might be helpful like perf traces and reports.
  Rodrigo.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1391339/+subscriptions