yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1906487] [NEW] [L2][scale] add a trunk size config option for bundle flow install

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: LIU Yulong <1906487@xxxxxxxxxxxxxxxxxx>
Date: Wed, 02 Dec 2020 08:47:03 -0000
Reply-to: Bug 1906487 <1906487@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

We recently tested the openvswitch-agent flow installation performance,
we have following two results:

1. the AGENT_RES_PROCESSING_STEP is 100 (default)
We can see the 4296 OpenFlow rules are installed in 20s.

2020-12-02 16:01:32.843 792 INFO neutron.agent.common.ovs_lib [None req-106cf38e-e20a-443e-b1ad-24ab1b7dc7cb - - - - -] Processing 4296 OpenFlow rules.
2020-12-02 16:01:52.698 792 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-106cf38e-e20a-443e-b1ad-24ab1b7dc7cb - - - - -] process_network_ports - iteration:266 - agent port security group processed in 31.044

2020-12-02T08:01:32.303Z|00967|connmgr|INFO|br-int<->unix#440: 1 flow_mods in the last 0 s (1 adds)
2020-12-02T08:01:33.353Z|00968|connmgr|INFO|br-int<->unix#443: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:33.864Z|00969|connmgr|INFO|br-int<->unix#446: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:34.435Z|00970|connmgr|INFO|br-int<->unix#449: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:34.966Z|00971|connmgr|INFO|br-int<->unix#452: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:35.399Z|00972|connmgr|INFO|br-int<->unix#455: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:35.904Z|00973|connmgr|INFO|br-int<->unix#458: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:36.335Z|00974|connmgr|INFO|br-int<->unix#461: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:36.820Z|00975|connmgr|INFO|br-int<->unix#464: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:37.288Z|00976|connmgr|INFO|br-int<->unix#467: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:37.805Z|00977|connmgr|INFO|br-int<->unix#470: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:38.329Z|00978|connmgr|INFO|br-int<->unix#473: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:38.792Z|00979|connmgr|INFO|br-int<->unix#476: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:39.248Z|00980|connmgr|INFO|br-int<->unix#479: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:39.687Z|00981|connmgr|INFO|br-int<->unix#482: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:40.164Z|00982|connmgr|INFO|br-int<->unix#485: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:40.568Z|00983|connmgr|INFO|br-int<->unix#488: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:40.978Z|00984|connmgr|INFO|br-int<->unix#491: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:41.453Z|00985|connmgr|INFO|br-int<->unix#494: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:41.892Z|00986|connmgr|INFO|br-int<->unix#497: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:42.264Z|00987|connmgr|INFO|br-int<->unix#500: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:42.639Z|00988|connmgr|INFO|br-int<->unix#503: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:43.077Z|00989|connmgr|INFO|br-int<->unix#506: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:43.452Z|00990|connmgr|INFO|br-int<->unix#509: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:43.879Z|00991|connmgr|INFO|br-int<->unix#512: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:44.385Z|00992|connmgr|INFO|br-int<->unix#515: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:44.821Z|00993|connmgr|INFO|br-int<->unix#518: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:45.299Z|00994|connmgr|INFO|br-int<->unix#521: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:45.809Z|00995|connmgr|INFO|br-int<->unix#524: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:46.310Z|00996|connmgr|INFO|br-int<->unix#527: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:46.805Z|00997|connmgr|INFO|br-int<->unix#530: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:47.185Z|00998|connmgr|INFO|br-int<->unix#533: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:47.567Z|00999|connmgr|INFO|br-int<->unix#536: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:47.987Z|01000|connmgr|INFO|br-int<->unix#539: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:48.417Z|01001|connmgr|INFO|br-int<->unix#542: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:48.865Z|01002|connmgr|INFO|br-int<->unix#545: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:49.408Z|01003|connmgr|INFO|br-int<->unix#548: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:49.918Z|01004|connmgr|INFO|br-int<->unix#551: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:50.399Z|01005|connmgr|INFO|br-int<->unix#554: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:50.871Z|01006|connmgr|INFO|br-int<->unix#557: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:51.337Z|01007|connmgr|INFO|br-int<->unix#560: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:51.627Z|01008|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: 16 flow_mods in the 7 s starting 27 s ago (12 adds, 4 deletes)
2020-12-02T08:01:51.771Z|01009|connmgr|INFO|br-int<->unix#563: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:52.285Z|01010|connmgr|INFO|br-int<->unix#566: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:52.695Z|01011|connmgr|INFO|br-int<->unix#569: 96 flow_mods in the last 0 s (96 adds)

2. the AGENT_RES_PROCESSING_STEP is 1000 (changed)
12222 OpenFlow rules are installed in 6s.

2020-12-02 16:13:41.855 3090 INFO neutron.agent.common.ovs_lib [None req-0eef620d-613e-4485-ad2f-b9380c3fc381 - - - - -] Processing 12222 OpenFlow rules.
2020-12-02 16:13:47.846 3090 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-0eef620d-613e-4485-ad2f-b9380c3fc381 - - - - -] process_network_ports - iteration:0 - agent port security group processed in 202.523

2020-12-02T08:13:42.299Z|01310|connmgr|INFO|br-int<->unix#1298: 1000 flow_mods in the last 0 s (1000 adds)
2020-12-02T08:13:42.718Z|01311|connmgr|INFO|br-int<->unix#1301: 1000 flow_mods in the last 0 s (1000 adds)
2020-12-02T08:13:43.174Z|01312|connmgr|INFO|br-int<->unix#1304: 1000 flow_mods in the last 0 s (1000 adds)
2020-12-02T08:13:43.698Z|01313|connmgr|INFO|br-int<->unix#1307: 1000 flow_mods in the last 0 s (1000 adds)
2020-12-02T08:13:44.087Z|01314|connmgr|INFO|br-int<->unix#1310: 1000 flow_mods in the last 0 s (1000 adds)
2020-12-02T08:13:44.554Z|01315|connmgr|INFO|br-int<->unix#1313: 1000 flow_mods in the last 0 s (1000 adds)
2020-12-02T08:13:45.050Z|01316|connmgr|INFO|br-int<->unix#1316: 1000 flow_mods in the last 0 s (1000 adds)
2020-12-02T08:13:45.534Z|01317|connmgr|INFO|br-int<->unix#1319: 1000 flow_mods in the last 0 s (1000 adds)
2020-12-02T08:13:45.939Z|01318|connmgr|INFO|br-int<->unix#1322: 1000 flow_mods in the last 0 s (1000 adds)
2020-12-02T08:13:46.388Z|01319|connmgr|INFO|br-int<->unix#1325: 1000 flow_mods in the last 0 s (1000 adds)
2020-12-02T08:13:46.869Z|01320|connmgr|INFO|br-int<->unix#1328: 1000 flow_mods in the last 0 s (1000 adds)
2020-12-02T08:13:47.323Z|01321|connmgr|INFO|br-int<->unix#1331: 1000 flow_mods in the last 0 s (1000 adds)
2020-12-02T08:13:47.834Z|01322|connmgr|INFO|br-int<->unix#1334: 222 flow_mods in the last 0 s (222 adds)

So we can say maybe we should increase the trunk (step) size for the
ovs-ofctl bundle installaion. We do not want to add a fixed value
because the vswitchd may be in high pressure according to your
deployment, the flow install performace may not be as good as you
expected. So we could add a config option for users, they can tune the
size for their cloud based on the load of the agent, vswitchd and cpu
usage and so on.

** Affects: neutron
     Importance: Undecided
     Assignee: LIU Yulong (dragon889)
         Status: New

** Description changed:

  We recently tested the openvswitch-agent flow installation performance,
  we have following two results:
  
  1. the AGENT_RES_PROCESSING_STEP is 100 (default)
  We can see the 4296 OpenFlow rules are installed in 20s.
  
- 
- 2020-12-02 16:01:32.843 792 INFO neutron.agent.common.ovs_lib [None req-106cf38e-e20a-443e-b1ad-24ab1b7dc7cb - - - - -] Processing 4296 OpenFlow rules. 
+ 2020-12-02 16:01:32.843 792 INFO neutron.agent.common.ovs_lib [None req-106cf38e-e20a-443e-b1ad-24ab1b7dc7cb - - - - -] Processing 4296 OpenFlow rules.
  2020-12-02 16:01:52.698 792 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-106cf38e-e20a-443e-b1ad-24ab1b7dc7cb - - - - -] process_network_ports - iteration:266 - agent port security group processed in 31.044
  
  2020-12-02T08:01:32.303Z|00967|connmgr|INFO|br-int<->unix#440: 1 flow_mods in the last 0 s (1 adds)
  2020-12-02T08:01:33.353Z|00968|connmgr|INFO|br-int<->unix#443: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:33.864Z|00969|connmgr|INFO|br-int<->unix#446: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:34.435Z|00970|connmgr|INFO|br-int<->unix#449: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:34.966Z|00971|connmgr|INFO|br-int<->unix#452: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:35.399Z|00972|connmgr|INFO|br-int<->unix#455: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:35.904Z|00973|connmgr|INFO|br-int<->unix#458: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:36.335Z|00974|connmgr|INFO|br-int<->unix#461: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:36.820Z|00975|connmgr|INFO|br-int<->unix#464: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:37.288Z|00976|connmgr|INFO|br-int<->unix#467: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:37.805Z|00977|connmgr|INFO|br-int<->unix#470: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:38.329Z|00978|connmgr|INFO|br-int<->unix#473: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:38.792Z|00979|connmgr|INFO|br-int<->unix#476: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:39.248Z|00980|connmgr|INFO|br-int<->unix#479: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:39.687Z|00981|connmgr|INFO|br-int<->unix#482: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:40.164Z|00982|connmgr|INFO|br-int<->unix#485: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:40.568Z|00983|connmgr|INFO|br-int<->unix#488: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:40.978Z|00984|connmgr|INFO|br-int<->unix#491: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:41.453Z|00985|connmgr|INFO|br-int<->unix#494: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:41.892Z|00986|connmgr|INFO|br-int<->unix#497: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:42.264Z|00987|connmgr|INFO|br-int<->unix#500: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:42.639Z|00988|connmgr|INFO|br-int<->unix#503: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:43.077Z|00989|connmgr|INFO|br-int<->unix#506: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:43.452Z|00990|connmgr|INFO|br-int<->unix#509: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:43.879Z|00991|connmgr|INFO|br-int<->unix#512: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:44.385Z|00992|connmgr|INFO|br-int<->unix#515: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:44.821Z|00993|connmgr|INFO|br-int<->unix#518: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:45.299Z|00994|connmgr|INFO|br-int<->unix#521: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:45.809Z|00995|connmgr|INFO|br-int<->unix#524: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:46.310Z|00996|connmgr|INFO|br-int<->unix#527: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:46.805Z|00997|connmgr|INFO|br-int<->unix#530: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:47.185Z|00998|connmgr|INFO|br-int<->unix#533: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:47.567Z|00999|connmgr|INFO|br-int<->unix#536: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:47.987Z|01000|connmgr|INFO|br-int<->unix#539: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:48.417Z|01001|connmgr|INFO|br-int<->unix#542: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:48.865Z|01002|connmgr|INFO|br-int<->unix#545: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:49.408Z|01003|connmgr|INFO|br-int<->unix#548: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:49.918Z|01004|connmgr|INFO|br-int<->unix#551: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:50.399Z|01005|connmgr|INFO|br-int<->unix#554: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:50.871Z|01006|connmgr|INFO|br-int<->unix#557: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:51.337Z|01007|connmgr|INFO|br-int<->unix#560: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:51.627Z|01008|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: 16 flow_mods in the 7 s starting 27 s ago (12 adds, 4 deletes)
  2020-12-02T08:01:51.771Z|01009|connmgr|INFO|br-int<->unix#563: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:52.285Z|01010|connmgr|INFO|br-int<->unix#566: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:52.695Z|01011|connmgr|INFO|br-int<->unix#569: 96 flow_mods in the last 0 s (96 adds)
  
- 
  2. the AGENT_RES_PROCESSING_STEP is 1000 (changed)
  12222 OpenFlow rules are installed in 6s.
- 
  
  2020-12-02 16:13:41.855 3090 INFO neutron.agent.common.ovs_lib [None req-0eef620d-613e-4485-ad2f-b9380c3fc381 - - - - -] Processing 12222 OpenFlow rules.
  2020-12-02 16:13:47.846 3090 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-0eef620d-613e-4485-ad2f-b9380c3fc381 - - - - -] process_network_ports - iteration:0 - agent port security group processed in 202.523
  
  2020-12-02T08:13:42.299Z|01310|connmgr|INFO|br-int<->unix#1298: 1000 flow_mods in the last 0 s (1000 adds)
  2020-12-02T08:13:42.718Z|01311|connmgr|INFO|br-int<->unix#1301: 1000 flow_mods in the last 0 s (1000 adds)
  2020-12-02T08:13:43.174Z|01312|connmgr|INFO|br-int<->unix#1304: 1000 flow_mods in the last 0 s (1000 adds)
  2020-12-02T08:13:43.698Z|01313|connmgr|INFO|br-int<->unix#1307: 1000 flow_mods in the last 0 s (1000 adds)
  2020-12-02T08:13:44.087Z|01314|connmgr|INFO|br-int<->unix#1310: 1000 flow_mods in the last 0 s (1000 adds)
  2020-12-02T08:13:44.554Z|01315|connmgr|INFO|br-int<->unix#1313: 1000 flow_mods in the last 0 s (1000 adds)
  2020-12-02T08:13:45.050Z|01316|connmgr|INFO|br-int<->unix#1316: 1000 flow_mods in the last 0 s (1000 adds)
  2020-12-02T08:13:45.534Z|01317|connmgr|INFO|br-int<->unix#1319: 1000 flow_mods in the last 0 s (1000 adds)
  2020-12-02T08:13:45.939Z|01318|connmgr|INFO|br-int<->unix#1322: 1000 flow_mods in the last 0 s (1000 adds)
  2020-12-02T08:13:46.388Z|01319|connmgr|INFO|br-int<->unix#1325: 1000 flow_mods in the last 0 s (1000 adds)
  2020-12-02T08:13:46.869Z|01320|connmgr|INFO|br-int<->unix#1328: 1000 flow_mods in the last 0 s (1000 adds)
  2020-12-02T08:13:47.323Z|01321|connmgr|INFO|br-int<->unix#1331: 1000 flow_mods in the last 0 s (1000 adds)
  2020-12-02T08:13:47.834Z|01322|connmgr|INFO|br-int<->unix#1334: 222 flow_mods in the last 0 s (222 adds)
  
- 
- So we can say maybe we can increase the trunk (step) size for the ovs-ofctl bundle installaion.
- We do not want to add a fixed value because the vswithd may be in high pressure according to your deployment, the flow install performace may not be as good as your excepted. So we can add a config option for users, they can tune the size for their cloud based on the load of the agent, vswitchd and cpu usage and so on.
+ So we can say maybe we should increase the trunk (step) size for the
+ ovs-ofctl bundle installaion. We do not want to add a fixed value
+ because the vswitchd may be in high pressure according to your
+ deployment, the flow install performace may not be as good as you
+ expected. So we could add a config option for users, they can tune the
+ size for their cloud based on the load of the agent, vswitchd and cpu
+ usage and so on.

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1906487

Title:
  [L2][scale] add a trunk size config option for bundle flow install

Status in neutron:
  New

Bug description:
  We recently tested the openvswitch-agent flow installation
  performance, we have following two results:

  1. the AGENT_RES_PROCESSING_STEP is 100 (default)
  We can see the 4296 OpenFlow rules are installed in 20s.

  2020-12-02 16:01:32.843 792 INFO neutron.agent.common.ovs_lib [None req-106cf38e-e20a-443e-b1ad-24ab1b7dc7cb - - - - -] Processing 4296 OpenFlow rules.
  2020-12-02 16:01:52.698 792 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-106cf38e-e20a-443e-b1ad-24ab1b7dc7cb - - - - -] process_network_ports - iteration:266 - agent port security group processed in 31.044

  2020-12-02T08:01:32.303Z|00967|connmgr|INFO|br-int<->unix#440: 1 flow_mods in the last 0 s (1 adds)
  2020-12-02T08:01:33.353Z|00968|connmgr|INFO|br-int<->unix#443: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:33.864Z|00969|connmgr|INFO|br-int<->unix#446: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:34.435Z|00970|connmgr|INFO|br-int<->unix#449: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:34.966Z|00971|connmgr|INFO|br-int<->unix#452: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:35.399Z|00972|connmgr|INFO|br-int<->unix#455: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:35.904Z|00973|connmgr|INFO|br-int<->unix#458: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:36.335Z|00974|connmgr|INFO|br-int<->unix#461: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:36.820Z|00975|connmgr|INFO|br-int<->unix#464: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:37.288Z|00976|connmgr|INFO|br-int<->unix#467: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:37.805Z|00977|connmgr|INFO|br-int<->unix#470: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:38.329Z|00978|connmgr|INFO|br-int<->unix#473: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:38.792Z|00979|connmgr|INFO|br-int<->unix#476: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:39.248Z|00980|connmgr|INFO|br-int<->unix#479: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:39.687Z|00981|connmgr|INFO|br-int<->unix#482: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:40.164Z|00982|connmgr|INFO|br-int<->unix#485: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:40.568Z|00983|connmgr|INFO|br-int<->unix#488: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:40.978Z|00984|connmgr|INFO|br-int<->unix#491: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:41.453Z|00985|connmgr|INFO|br-int<->unix#494: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:41.892Z|00986|connmgr|INFO|br-int<->unix#497: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:42.264Z|00987|connmgr|INFO|br-int<->unix#500: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:42.639Z|00988|connmgr|INFO|br-int<->unix#503: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:43.077Z|00989|connmgr|INFO|br-int<->unix#506: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:43.452Z|00990|connmgr|INFO|br-int<->unix#509: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:43.879Z|00991|connmgr|INFO|br-int<->unix#512: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:44.385Z|00992|connmgr|INFO|br-int<->unix#515: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:44.821Z|00993|connmgr|INFO|br-int<->unix#518: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:45.299Z|00994|connmgr|INFO|br-int<->unix#521: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:45.809Z|00995|connmgr|INFO|br-int<->unix#524: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:46.310Z|00996|connmgr|INFO|br-int<->unix#527: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:46.805Z|00997|connmgr|INFO|br-int<->unix#530: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:47.185Z|00998|connmgr|INFO|br-int<->unix#533: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:47.567Z|00999|connmgr|INFO|br-int<->unix#536: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:47.987Z|01000|connmgr|INFO|br-int<->unix#539: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:48.417Z|01001|connmgr|INFO|br-int<->unix#542: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:48.865Z|01002|connmgr|INFO|br-int<->unix#545: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:49.408Z|01003|connmgr|INFO|br-int<->unix#548: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:49.918Z|01004|connmgr|INFO|br-int<->unix#551: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:50.399Z|01005|connmgr|INFO|br-int<->unix#554: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:50.871Z|01006|connmgr|INFO|br-int<->unix#557: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:51.337Z|01007|connmgr|INFO|br-int<->unix#560: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:51.627Z|01008|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: 16 flow_mods in the 7 s starting 27 s ago (12 adds, 4 deletes)
  2020-12-02T08:01:51.771Z|01009|connmgr|INFO|br-int<->unix#563: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:52.285Z|01010|connmgr|INFO|br-int<->unix#566: 100 flow_mods in the last 0 s (100 adds)
  2020-12-02T08:01:52.695Z|01011|connmgr|INFO|br-int<->unix#569: 96 flow_mods in the last 0 s (96 adds)

  2. the AGENT_RES_PROCESSING_STEP is 1000 (changed)
  12222 OpenFlow rules are installed in 6s.

  2020-12-02 16:13:41.855 3090 INFO neutron.agent.common.ovs_lib [None req-0eef620d-613e-4485-ad2f-b9380c3fc381 - - - - -] Processing 12222 OpenFlow rules.
  2020-12-02 16:13:47.846 3090 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-0eef620d-613e-4485-ad2f-b9380c3fc381 - - - - -] process_network_ports - iteration:0 - agent port security group processed in 202.523

  2020-12-02T08:13:42.299Z|01310|connmgr|INFO|br-int<->unix#1298: 1000 flow_mods in the last 0 s (1000 adds)
  2020-12-02T08:13:42.718Z|01311|connmgr|INFO|br-int<->unix#1301: 1000 flow_mods in the last 0 s (1000 adds)
  2020-12-02T08:13:43.174Z|01312|connmgr|INFO|br-int<->unix#1304: 1000 flow_mods in the last 0 s (1000 adds)
  2020-12-02T08:13:43.698Z|01313|connmgr|INFO|br-int<->unix#1307: 1000 flow_mods in the last 0 s (1000 adds)
  2020-12-02T08:13:44.087Z|01314|connmgr|INFO|br-int<->unix#1310: 1000 flow_mods in the last 0 s (1000 adds)
  2020-12-02T08:13:44.554Z|01315|connmgr|INFO|br-int<->unix#1313: 1000 flow_mods in the last 0 s (1000 adds)
  2020-12-02T08:13:45.050Z|01316|connmgr|INFO|br-int<->unix#1316: 1000 flow_mods in the last 0 s (1000 adds)
  2020-12-02T08:13:45.534Z|01317|connmgr|INFO|br-int<->unix#1319: 1000 flow_mods in the last 0 s (1000 adds)
  2020-12-02T08:13:45.939Z|01318|connmgr|INFO|br-int<->unix#1322: 1000 flow_mods in the last 0 s (1000 adds)
  2020-12-02T08:13:46.388Z|01319|connmgr|INFO|br-int<->unix#1325: 1000 flow_mods in the last 0 s (1000 adds)
  2020-12-02T08:13:46.869Z|01320|connmgr|INFO|br-int<->unix#1328: 1000 flow_mods in the last 0 s (1000 adds)
  2020-12-02T08:13:47.323Z|01321|connmgr|INFO|br-int<->unix#1331: 1000 flow_mods in the last 0 s (1000 adds)
  2020-12-02T08:13:47.834Z|01322|connmgr|INFO|br-int<->unix#1334: 222 flow_mods in the last 0 s (222 adds)

  So we can say maybe we should increase the trunk (step) size for the
  ovs-ofctl bundle installaion. We do not want to add a fixed value
  because the vswitchd may be in high pressure according to your
  deployment, the flow install performace may not be as good as you
  expected. So we could add a config option for users, they can tune the
  size for their cloud based on the load of the agent, vswitchd and cpu
  usage and so on.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1906487/+subscriptions