yahoo-eng-team team mailing list archive

[OpenStack Infra <1866039@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/711048
Committed: https://git.openstack.org/cgit/openstack/neutron-tempest-plugin/commit/?id=41b8019c7b4b3921a077a032463b9d8c74957b4b
Submitter: Zuul
Branch:    master

commit 41b8019c7b4b3921a077a032463b9d8c74957b4b
Author: Maciej Józefczyk <mjozefcz@xxxxxxxxxx>
Date:   Tue Mar 3 17:10:57 2020 +0100

    QoS - Change the way we measure bw limits
    
    This patch introduces new way of fetching the data.
    Instead creating the file, it reads /dev/zero.
    /dev/zero is always very fast, so we also break the
    previous hard disk limitations.
    
    The test time is limited to 5 seconds. After that we
    calculate avg bytes per second value and compare it
    to expected one.
    
    Sometimes it is visible that first kilobytes of the
    test file are downloaded a little bit faster than the
    actual bw limit claims, especially while testing OVN
    as a backend.
    When it happens the avg bytes per second value that is
    measured in the test could be higher than required limit.
    
    It is pretty easy to show the case while testing QoS with iperf3:
    
    Accepted connection from 172.24.5.1, port 59690
    [  5] local 10.1.0.35 port 5201 connected to 172.24.5.1 port 59692
    [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
    [  5]   0.00-1.00   sec  1.32 MBytes  11.0 Mbits/sec  139   2.62 KBytes
    [  5]   1.00-2.00   sec   628 KBytes  5.15 Mbits/sec   96   10.5 KBytes
    [  5]   2.00-3.00   sec   502 KBytes  4.12 Mbits/sec   84   7.85 KBytes
    [  5]   3.00-4.00   sec   649 KBytes  5.32 Mbits/sec   83   10.5 KBytes
    [  5]   4.00-5.00   sec   643 KBytes  5.26 Mbits/sec   84   3.93 KBytes
    [  5]   5.00-6.00   sec   529 KBytes  4.33 Mbits/sec   73   5.23 KBytes
    [  5]   6.00-7.00   sec   628 KBytes  5.15 Mbits/sec   92   20.9 KBytes
    [  5]   7.00-8.00   sec   534 KBytes  4.37 Mbits/sec   82   18.3 KBytes
    [  5]   8.00-9.00   sec   667 KBytes  5.47 Mbits/sec  110   7.85 KBytes
    [  5]   9.00-10.00  sec   635 KBytes  5.20 Mbits/sec   90   11.8 KBytes
    [  5]  10.00-10.02  sec  0.00 Bytes  0.00 bits/sec    0   11.8 KBytes
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval           Transfer     Bandwidth       Retr
    [  5]   0.00-10.02  sec  6.61 MBytes  5.53 Mbits/sec  933             sender
    [  5]   0.00-10.02  sec  6.41 MBytes  5.36 Mbits/sec                  receiver
    -----------------------------------------------------------
    
    We can find out that during first second of the test the bw limit
    is exceeded, but after that the traffic is shaped.
    
    In our case when we run the tempest QoS test the avg bytes per second
    measured value that we compare with bw limit is impacted.
    
    Closes-Bug: 1866039
    
    Change-Id: I0964464e709baf9958548384933bd000fdee979b


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1866039

Title:
  [OVN] QoS gives different bandwidth limit measures than ml2/ovs

Status in neutron:
  Fix Released

Bug description:
  There is a difference in QoS tempest tests results between ml2/ovs and
  ml2/ovn.

  In the change [1] that enables QoS tempest tests for OVN the test neutron_tempest_plugin.scenario.test_qos.QoSTest.test_qos_basic_and_update
  fails on the last check [2], after the policy is updated to be configured with values:

  max_kbps=constants.LIMIT_KILO_BITS_PER_SECOND * 3
  max_burst_kbps=constants.LIMIT_KILO_BITS_PER_SECOND * 3, 

  Which means:
  max_kbps = 3000
  max_burst_kbps = 3000

  Previous QoS validations in this test passes with values (max_kbps,
  max_burst_kbps): (1000, 1000) and (2000, 2000).

  I added some more debug log to the tempest test here [3], so that we
  can compare test expected and measured values. Those are taken from
  test runs from gates.

  
  -----------------------------------------------------------------------
  Expected is calculated as:
  TOLERANCE_FACTOR = 1.5
  constants.LIMIT_KILO_BITS_PER_SECOND = 1000
  MULTIPLEXING_FACTOR = 1 or 2 or 3 depends on stage of the test

      LIMIT_BYTES_SEC = (constants.LIMIT_KILO_BITS_PER_SECOND * 1024 *
                         TOLERANCE_FACTOR / 8.0) * MULTIPLEXING_FACTOR
  -----------------------------------------------------------------------
  Results:
  If expected <= measured, the test passes.

  |max_kbps/max_burst_kbps|expected(bps)|ovs(bps)|ovn(bps)|linux_bridge(bps)|
  |(1000, 1000)|192000|112613|141250|129124|
  |(2000, 2000)|384000|311978|408886, 411005, 385152, 422114, 352903|300163|
  |(3000, 3000)|576000|523677|820522,..... failed|459569|

  As we see only for (3000, 3000) OVN test failed. For (2000, 2000) it
  passed after 5 retries.

  -----------------------------------------------------------------------

  So lets see how the QoS is configured on OVN nowadays:

  stack@mjozefcz-devstack-qos-2:~/logs$ neutron qos-bandwidth-limit-rule-list  047f7a8c-e143-471f-979c-4a4d95cefa5e 
  neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
  +-----------+--------------------------------------+----------------+----------+
  | direction | id                                   | max_burst_kbps | max_kbps |
  +-----------+--------------------------------------+----------------+----------+
  | egress    | 9dd84dc7-f216-432f-b1aa-ec17eb488720 |           3000 |     3000 |
  +-----------+--------------------------------------+----------------+----------+

  
  Configured OVN NBDB:
  stack@mjozefcz-devstack-qos-2:~/logs$ ovn-nbctl list qos
  _uuid               : 1176fe8f-695d-4f79-a99f-f0df8a7b8652
  action              : {}
  bandwidth           : {burst=3000, rate=3000}
  direction           : from-lport
  external_ids        : {}
  match               : "inport == \"4521ef05-d139-4d84-a100-efb83fde2b47\""
  priority            : 2002

  Configured meter on bridge:
  stack@mjozefcz-devstack-qos-2:~/logs$ sudo ovs-ofctl -O OpenFlow13  dump-meters br-int
  OFPST_METER_CONFIG reply (OF1.3) (xid=0x2):
  meter=1 kbps burst stats bands=
  type=drop rate=3000 burst_size=3000

  
  Flow in bridge:
  stack@mjozefcz-devstack-qos-2:~/logs$ sudo ovs-ofctl -O OpenFlow13 dump-flows br-int | grep meter
   cookie=0x398f0e17, duration=71156.273s, table=16, n_packets=136127, n_bytes=41572857, priority=2002,reg14=0x4,metadata=0x1 actions=meter:1,resubmit(,17)

  
  --------------------------------------------------------------------------

  Questions:
  * Why the test results are different compared to ml2/OVS?
  * Maybe burst values should be configured differently?

  
  [1] https://review.opendev.org/#/c/704833/
  [2] https://github.com/openstack/neutron-tempest-plugin/blob/328edc882a3debf4f1b39687dfb559d7c5c385f3/neutron_tempest_plugin/scenario/test_qos.py#L271
  [3] https://review.opendev.org/#/c/711048/

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1866039/+subscriptions