yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1866039] [NEW] [OVN] QoS gives different bandwidth limit measures than ml2/ovs

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Maciej Jozefczyk <1866039@xxxxxxxxxxxxxxxxxx>
Date: Wed, 04 Mar 2020 10:40:45 -0000
Reply-to: Bug 1866039 <1866039@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

There is a difference in QoS tempest tests results between ml2/ovs and
ml2/ovn.

In the change [1] that enables QoS tempest tests for OVN the test neutron_tempest_plugin.scenario.test_qos.QoSTest.test_qos_basic_and_update
fails on the last check [2], after the policy is updated to be configured with values:

max_kbps=constants.LIMIT_KILO_BITS_PER_SECOND * 3
max_burst_kbps=constants.LIMIT_KILO_BITS_PER_SECOND * 3, 

Which means:
max_kbps = 3000
max_burst_kbps = 3000

Previous QoS validations in this test passes with values (max_kbps,
max_burst_kbps): (1000, 1000) and (2000, 2000).

I added some more debug log to the tempest test here [3], so that we can
compare test expected and measured values. Those are taken from test
runs from gates.


-----------------------------------------------------------------------
Expected is calculated as:
TOLERANCE_FACTOR = 1.5
constants.LIMIT_KILO_BITS_PER_SECOND = 1000
MULTIPLEXING_FACTOR = 1 or 2 or 3 depends on stage of the test

    LIMIT_BYTES_SEC = (constants.LIMIT_KILO_BITS_PER_SECOND * 1024 *
                       TOLERANCE_FACTOR / 8.0) * MULTIPLEXING_FACTOR
-----------------------------------------------------------------------
Results:
If expected <= measured, the test passes.

|max_kbps/max_burst_kbps|expected(bps)|ovs(bps)|ovn(bps)|linux_bridge(bps)|
|(1000, 1000)|192000|112613|141250|129124|
|(2000, 2000)|384000|311978|408886, 411005, 385152, 422114, 352903|300163|
|(3000, 3000)|576000|523677|820522,..... failed|459569|

As we see only for (3000, 3000) OVN test failed. For (2000, 2000) it
passed after 5 retries.

-----------------------------------------------------------------------

So lets see how the QoS is configured on OVN nowadays:

stack@mjozefcz-devstack-qos-2:~/logs$ neutron qos-bandwidth-limit-rule-list  047f7a8c-e143-471f-979c-4a4d95cefa5e 
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
+-----------+--------------------------------------+----------------+----------+
| direction | id                                   | max_burst_kbps | max_kbps |
+-----------+--------------------------------------+----------------+----------+
| egress    | 9dd84dc7-f216-432f-b1aa-ec17eb488720 |           3000 |     3000 |
+-----------+--------------------------------------+----------------+----------+


Configured OVN NBDB:
stack@mjozefcz-devstack-qos-2:~/logs$ ovn-nbctl list qos
_uuid               : 1176fe8f-695d-4f79-a99f-f0df8a7b8652
action              : {}
bandwidth           : {burst=3000, rate=3000}
direction           : from-lport
external_ids        : {}
match               : "inport == \"4521ef05-d139-4d84-a100-efb83fde2b47\""
priority            : 2002

Configured meter on bridge:
stack@mjozefcz-devstack-qos-2:~/logs$ sudo ovs-ofctl -O OpenFlow13  dump-meters br-int
OFPST_METER_CONFIG reply (OF1.3) (xid=0x2):
meter=1 kbps burst stats bands=
type=drop rate=3000 burst_size=3000


Flow in bridge:
stack@mjozefcz-devstack-qos-2:~/logs$ sudo ovs-ofctl -O OpenFlow13 dump-flows br-int | grep meter
 cookie=0x398f0e17, duration=71156.273s, table=16, n_packets=136127, n_bytes=41572857, priority=2002,reg14=0x4,metadata=0x1 actions=meter:1,resubmit(,17)


--------------------------------------------------------------------------

Questions:
* Why the test results are different compared to ml2/OVS?
* Maybe burst values should be configured differently?


[1] https://review.opendev.org/#/c/704833/
[2] https://github.com/openstack/neutron-tempest-plugin/blob/328edc882a3debf4f1b39687dfb559d7c5c385f3/neutron_tempest_plugin/scenario/test_qos.py#L271
[3] https://review.opendev.org/#/c/711048/

** Affects: neutron
     Importance: Undecided
     Assignee: Maciej Jozefczyk (maciej.jozefczyk)
         Status: New


** Tags: ovn qos

** Changed in: neutron
     Assignee: (unassigned) => Maciej Jozefczyk (maciej.jozefczyk)

** Summary changed:

- [OVN] QoS gives different burst limit values
+ [OVN] QoS gives different bandwidth limit values

** Summary changed:

- [OVN] QoS gives different bandwidth limit values
+ [OVN] QoS gives different bandwidth limit measures than ml2/ovs

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1866039

Title:
  [OVN] QoS gives different bandwidth limit measures than ml2/ovs

Status in neutron:
  New

Bug description:
  There is a difference in QoS tempest tests results between ml2/ovs and
  ml2/ovn.

  In the change [1] that enables QoS tempest tests for OVN the test neutron_tempest_plugin.scenario.test_qos.QoSTest.test_qos_basic_and_update
  fails on the last check [2], after the policy is updated to be configured with values:

  max_kbps=constants.LIMIT_KILO_BITS_PER_SECOND * 3
  max_burst_kbps=constants.LIMIT_KILO_BITS_PER_SECOND * 3, 

  Which means:
  max_kbps = 3000
  max_burst_kbps = 3000

  Previous QoS validations in this test passes with values (max_kbps,
  max_burst_kbps): (1000, 1000) and (2000, 2000).

  I added some more debug log to the tempest test here [3], so that we
  can compare test expected and measured values. Those are taken from
  test runs from gates.

  
  -----------------------------------------------------------------------
  Expected is calculated as:
  TOLERANCE_FACTOR = 1.5
  constants.LIMIT_KILO_BITS_PER_SECOND = 1000
  MULTIPLEXING_FACTOR = 1 or 2 or 3 depends on stage of the test

      LIMIT_BYTES_SEC = (constants.LIMIT_KILO_BITS_PER_SECOND * 1024 *
                         TOLERANCE_FACTOR / 8.0) * MULTIPLEXING_FACTOR
  -----------------------------------------------------------------------
  Results:
  If expected <= measured, the test passes.

  |max_kbps/max_burst_kbps|expected(bps)|ovs(bps)|ovn(bps)|linux_bridge(bps)|
  |(1000, 1000)|192000|112613|141250|129124|
  |(2000, 2000)|384000|311978|408886, 411005, 385152, 422114, 352903|300163|
  |(3000, 3000)|576000|523677|820522,..... failed|459569|

  As we see only for (3000, 3000) OVN test failed. For (2000, 2000) it
  passed after 5 retries.

  -----------------------------------------------------------------------

  So lets see how the QoS is configured on OVN nowadays:

  stack@mjozefcz-devstack-qos-2:~/logs$ neutron qos-bandwidth-limit-rule-list  047f7a8c-e143-471f-979c-4a4d95cefa5e 
  neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
  +-----------+--------------------------------------+----------------+----------+
  | direction | id                                   | max_burst_kbps | max_kbps |
  +-----------+--------------------------------------+----------------+----------+
  | egress    | 9dd84dc7-f216-432f-b1aa-ec17eb488720 |           3000 |     3000 |
  +-----------+--------------------------------------+----------------+----------+

  
  Configured OVN NBDB:
  stack@mjozefcz-devstack-qos-2:~/logs$ ovn-nbctl list qos
  _uuid               : 1176fe8f-695d-4f79-a99f-f0df8a7b8652
  action              : {}
  bandwidth           : {burst=3000, rate=3000}
  direction           : from-lport
  external_ids        : {}
  match               : "inport == \"4521ef05-d139-4d84-a100-efb83fde2b47\""
  priority            : 2002

  Configured meter on bridge:
  stack@mjozefcz-devstack-qos-2:~/logs$ sudo ovs-ofctl -O OpenFlow13  dump-meters br-int
  OFPST_METER_CONFIG reply (OF1.3) (xid=0x2):
  meter=1 kbps burst stats bands=
  type=drop rate=3000 burst_size=3000

  
  Flow in bridge:
  stack@mjozefcz-devstack-qos-2:~/logs$ sudo ovs-ofctl -O OpenFlow13 dump-flows br-int | grep meter
   cookie=0x398f0e17, duration=71156.273s, table=16, n_packets=136127, n_bytes=41572857, priority=2002,reg14=0x4,metadata=0x1 actions=meter:1,resubmit(,17)

  
  --------------------------------------------------------------------------

  Questions:
  * Why the test results are different compared to ml2/OVS?
  * Maybe burst values should be configured differently?

  
  [1] https://review.opendev.org/#/c/704833/
  [2] https://github.com/openstack/neutron-tempest-plugin/blob/328edc882a3debf4f1b39687dfb559d7c5c385f3/neutron_tempest_plugin/scenario/test_qos.py#L271
  [3] https://review.opendev.org/#/c/711048/

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1866039/+subscriptions
Follow ups

[Bug 1866039] Re: [OVN] QoS gives different bandwidth limit measures than ml2/ovs
From: OpenStack Infra, 2020-03-16