yahoo-eng-team team mailing list archive

[Ha Van Tu <tuhv@xxxxxxxxxxxxxx>]

Public bug reported:

Updating firewall with a large number of firewall rules needs improving
performance.

When the Firewall is updated, the conntrack entries will be deleted by conntrack-tools ("conntrack -D" commands) with each rule associated with this firewall. The problem is inside a cloud system with a large number of firewall rules applied. Updating so much rules will lead to call a large number of subprocesses to implement the "conntrack -D" commands. That will consume the system resource and  it will take a long time to finish updating firewall[1].
By using Netlink, we can call the subprocess only one time [6], so as to reduce the system resource and time to update firewall.

There should be some critical points need to be discussed:
- The standard Netlink interface for Python. There are 2 sources: [3] and [4] on github, but I don't know these resources are acceptable or not.
- The "conntrack -D" needs *root privilege*. My solution is make the Python module which performs deleting conntrack entries become Linux command (calling "python pythonmodule.py") and wrap by rootwrap.[5] 

[1] With the system with Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz and 16GiB memory, it take 429s to finish removing 10.000 rules. The client is in [2]
[2] http://paste.openstack.org/show/584602/
[3] https://github.com/ei-grad/python-conntrack
[4] https://github.com/regit/pynetfilter_conntrack
[5] https://ask.openstack.org/en/question/60893/rootwrap-python-write-to-root-only-owned-file/
[6] http://paste.openstack.org/show/584603/

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: fwaas needs-attention rfe

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1630832

Title:
  [RFE] FWaaS: Using Netlink instead of conntrack-tools to improve
  performance

Status in neutron:
  New

Bug description:
  Updating firewall with a large number of firewall rules needs
  improving performance.

  When the Firewall is updated, the conntrack entries will be deleted by conntrack-tools ("conntrack -D" commands) with each rule associated with this firewall. The problem is inside a cloud system with a large number of firewall rules applied. Updating so much rules will lead to call a large number of subprocesses to implement the "conntrack -D" commands. That will consume the system resource and  it will take a long time to finish updating firewall[1].
  By using Netlink, we can call the subprocess only one time [6], so as to reduce the system resource and time to update firewall.

  There should be some critical points need to be discussed:
  - The standard Netlink interface for Python. There are 2 sources: [3] and [4] on github, but I don't know these resources are acceptable or not.
  - The "conntrack -D" needs *root privilege*. My solution is make the Python module which performs deleting conntrack entries become Linux command (calling "python pythonmodule.py") and wrap by rootwrap.[5] 

  [1] With the system with Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz and 16GiB memory, it take 429s to finish removing 10.000 rules. The client is in [2]
  [2] http://paste.openstack.org/show/584602/
  [3] https://github.com/ei-grad/python-conntrack
  [4] https://github.com/regit/pynetfilter_conntrack
  [5] https://ask.openstack.org/en/question/60893/rootwrap-python-write-to-root-only-owned-file/
  [6] http://paste.openstack.org/show/584603/

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1630832/+subscriptions