openstack team mailing list archive

[Rick Jones <rick.jones2@xxxxxx>]

On 03/15/2013 08:05 AM, tommy(小包) wrote:
> Hi Guys，
>
> in my test, i found OVS GRE performance so lower, for example：
> 100Mbits Switch， GRE just 26Mbits speed，but use linux bridge
> 95Mbits,
>
> so, my question is: why GRE speed low, or may be my config not right,/

95 and 26 Mbit/s measured at what "level?"  On the wire (including all 
the protocol headers) or to user level (after all the protocol headers)? 
That you were seeing 95 Mbit/s suggests user level but I'd like to make 
certain.

GRE adds header overhead, but I wouldn't think enough to take one from 
95 down to 26 Mbit/s to user level.  I would suggest looking at, in no 
particular order:

*) Netstat stats on your sender - is it retransmitting in one case and 
not the other?

*) per-CPU CPU utilization - is any one CPU on the sending, receiving or 
intervening iron saturating in one case and not the other?

and go from there. I'm guessing your tests are all bulk-transfer - you 
might want to consider adding some latency and/or aggregate small-packet 
performance tests.

happy benchmarking,

rick jones
the applicability varies,  but attached is some boilerplate I've 
built-up over time, on the matter of "why is my network performance 
slow?"  PS - the "beforeafter" utility mentioned is no longer available 
via ftp.cup.hp.com because ftp.cup.hp.com no longer exists.  I probably 
aught to put it up on ftp.netperf.org...


Some of my checklist items when presented with assertions of poor
network performance, in no particular order, numbered only for
convenience of reference:

1) Is *any one* CPU on either end of the transfer at or close to 100%
   utilization?  A given TCP connection cannot really take advantage
   of more than the services of a single core in the system, so
   average CPU utilization being low does not a priori mean things are
   OK.

2) Are there TCP retransmissions being registered in netstat
   statistics on the sending system?  Take a snapshot of netstat -s -t
   from just before the transfer, and one from just after and run it
   through beforeafter from
   ftp://ftp.cup.hp.com/dist/networking/tools:

   netstat -s -t > before
   transfer or wait 60 or so seconds if the transfer was already going
   netstat -s -t > after
   beforeafter before after > delta

3) Are there packet drops registered in ethtool -S statistics on
   either side of the transfer?  Take snapshots in a manner similar to
   that with netstat.

4) Are there packet drops registered in the stats for the switch(es)
   being traversed by the transfer?  These would be retrieved via
   switch-specific means.

5) What is the latency between the two end points.  Install netperf on
   both sides, start netserver on one side and on the other side run:

   netperf -t TCP_RR -l 30 -H <remote>

   and invert the transaction/s rate to get the RTT latency.  There
   are caveats involving NIC interrupt coalescing settings defaulting
   in favor of throughput/CPU util over latency:

   ftp://ftp.cup.hp.com/dist/networking/briefs/nic_latency_vs_tput.txt

   but when the connections are over a WAN latency is important and
   may not be clouded as much by NIC settings.

   This all leads into:

6) What is the *effective* TCP (or other) window size for the
   connection.  One limit to the performance of a TCP bulk transfer
   is:

   Tput <= W(eff)/RTT

   The effective window size will be the lesser of:

   a) The classic TCP window advertised by the receiver. This is the
      value in the TCP header's window field shifted by the window
      scaling factor which was exchanged during connection
      establishment. The window scale factor is why one wants to get
      traces including the connection establishment.
   
      The size of the classic window will depend on whether/what the
      receiving application has requested via a setsockopt(SO_RCVBUF)
      call and the sysctl limits set in the OS.  If the receiving
      application does not call setsockopt(SO_RCVBUF) then under Linux
      the stack will "autotune" the advertised window based on other
      sysctl limits in the OS.  Other stacks may or may not autotune.
 
   b) The computed congestion window on the sender - this will be
      affected by the packet loss rate over the connection, hence the
      interest in the netstat and ethtool stats.

   c) The quantity of data to which the sending TCP can maintain a
      reference while waiting for it to be ACKnowledged by the
      receiver - this will be akin to the classic TCP window case
      above, but on the sending side, and concerning
      setsockopt(SO_SNDBUF) and sysctl settings.

   d) The quantity of data the sending application is willing/able to
      send at any one time before waiting for some sort of
      application-level acknowledgement.  FTP and rcp will just blast
      all the data of the file into the socket as fast as the socket
      will take it.  Scp has some application-layer "windowing" which
      may cause it to put less data out onto the connection than TCP
      might otherwise have permitted.  NFS has the maximum number of
      outstanding requests it will allow at one time acting as a
      defacto "window" etc etc etc

7) Another magic forumla for TCP bulk transfer performance comes from
   Mathis, Semke, Mahdavi & Ott
   http://www.psc.edu/networking/papers/model_ccr97.ps

   Tput <= (MSS/RTT) * (1/sqrt(p))

   MSS is Maximum Segment Size
   RTT is Round Trip Time
   p   is the packet loss rate as a probability (eg values of 0 to 1.0)

   Which assumes a few things about the congestion control algorithm
   being used and that there is no classic TCP window limitation as
   mentioned in item 6.

8) Is the link/path between the sender and the receiver composed of
   single-link hops, or might some be aggregated link hops?  If the
   latter, does traffic from a single flow (eg TCP connection) get
   striped across each link, or does it stay on just one link in the
   aggregation(s)?  Striping across multiple links can lead to packet
   re-ordering which will affect TCP performance.  If there are
   aggregated links and no striping then the advertised "N Gbit/s" may
   really be "1/n Gbit/s" per flow.