yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #43449
[Bug 1526967] [NEW] Use diffs for iptables restore instead of all rules
Public bug reported:
https://review.openstack.org/255334
Dear bug triager. This bug was created since a commit was marked with DOCIMPACT.
Your project "openstack/neutron" is set up so that we directly report the documentation bugs against it. If this needs changing, the docimpact-group option needs to be added for the project. You can ask the OpenStack infra team (#openstack-infra on freenode) for help if you need to.
commit fb55693713991782a56006af73e0ee67cffe9a56
Author: Kevin Benton <blak111@xxxxxxxxx>
Date: Sat Oct 3 07:25:19 2015 -0700
Use diffs for iptables restore instead of all rules
This patch changes our iptables logic to generate a delta of
iptables commands (inserts + deletes) to get from the current
iptables state to the new state. This will significantly reduce
the amount of data that we have to shell out to iptables-restore
on every call (and reduce the amount of data iptables-restore has
to parse).
We no longer have to worry about preserving counters since
we are adding and deleting specific rules, so the rule modification
code got a nice cleanup to get rid of the old rule matching.
This also gives us a new method of functionally testing that we are
generating rules in the correct manner. After applying new rules
once, a subsequent call should always have no work to do. The new
functional tests added leverage that property heavily and should
protect us from regressions in how rules are formed.
Performance metrics relative to HEAD~1:
+====================================+============+=======+
| Scenario | This patch | HEAD~1|
|------------------------------------|------------|-------|
| 200 VMs*22 rules existing - startup| | |
| _modify_rules| 0.67s | 1.05s |
| _apply_synchronized| 1.87s | 2.89s |
|------------------------------------|------------|-------|
| 200 VMs*22 rules existing - add VM | | |
| _modify_rules| 0.68s | 1.05s |
| _apply_synchronized| 2.07s | 2.92s |
|------------------------------------+------------+-------+
|200 VMs*422 rules existing - startup| | |
| _modify_rules| 5.43s | 8.17s |
| _apply_synchronized| 12.77s |28.00s |
|------------------------------------|------------|-------|
|200 VMs*422 rules existing - add VM | | |
| _modify_rules| 6.41s | 8.33s |
| _apply_synchronized| 33.09s |33.80s |
+------------------------------------+------------+-------+
The _apply_synchronized times seem to converge when dealing
with ~85k rules. In the profile I can see that both approaches
seem to wait on iptables-restore for approximately the same
amount of time so it could be hitting the performance limits
of iptables-restore.
DocImpact
Partial-Bug: #1502297
Change-Id: Ia6470c85b6b71979006ffe5da9095fdcce3122c1
(cherry picked from commit f066e46bb77362ccca838b541eb42c7ae8ddc125)
** Affects: neutron
Importance: Undecided
Status: New
** Tags: neutron
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1526967
Title:
Use diffs for iptables restore instead of all rules
Status in neutron:
New
Bug description:
https://review.openstack.org/255334
Dear bug triager. This bug was created since a commit was marked with DOCIMPACT.
Your project "openstack/neutron" is set up so that we directly report the documentation bugs against it. If this needs changing, the docimpact-group option needs to be added for the project. You can ask the OpenStack infra team (#openstack-infra on freenode) for help if you need to.
commit fb55693713991782a56006af73e0ee67cffe9a56
Author: Kevin Benton <blak111@xxxxxxxxx>
Date: Sat Oct 3 07:25:19 2015 -0700
Use diffs for iptables restore instead of all rules
This patch changes our iptables logic to generate a delta of
iptables commands (inserts + deletes) to get from the current
iptables state to the new state. This will significantly reduce
the amount of data that we have to shell out to iptables-restore
on every call (and reduce the amount of data iptables-restore has
to parse).
We no longer have to worry about preserving counters since
we are adding and deleting specific rules, so the rule modification
code got a nice cleanup to get rid of the old rule matching.
This also gives us a new method of functionally testing that we are
generating rules in the correct manner. After applying new rules
once, a subsequent call should always have no work to do. The new
functional tests added leverage that property heavily and should
protect us from regressions in how rules are formed.
Performance metrics relative to HEAD~1:
+====================================+============+=======+
| Scenario | This patch | HEAD~1|
|------------------------------------|------------|-------|
| 200 VMs*22 rules existing - startup| | |
| _modify_rules| 0.67s | 1.05s |
| _apply_synchronized| 1.87s | 2.89s |
|------------------------------------|------------|-------|
| 200 VMs*22 rules existing - add VM | | |
| _modify_rules| 0.68s | 1.05s |
| _apply_synchronized| 2.07s | 2.92s |
|------------------------------------+------------+-------+
|200 VMs*422 rules existing - startup| | |
| _modify_rules| 5.43s | 8.17s |
| _apply_synchronized| 12.77s |28.00s |
|------------------------------------|------------|-------|
|200 VMs*422 rules existing - add VM | | |
| _modify_rules| 6.41s | 8.33s |
| _apply_synchronized| 33.09s |33.80s |
+------------------------------------+------------+-------+
The _apply_synchronized times seem to converge when dealing
with ~85k rules. In the profile I can see that both approaches
seem to wait on iptables-restore for approximately the same
amount of time so it could be hitting the performance limits
of iptables-restore.
DocImpact
Partial-Bug: #1502297
Change-Id: Ia6470c85b6b71979006ffe5da9095fdcce3122c1
(cherry picked from commit f066e46bb77362ccca838b541eb42c7ae8ddc125)
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1526967/+subscriptions
