yahoo-eng-team team mailing list archive

[Li Ma <skywalker.nick@xxxxxxxxx>]

Public bug reported:

[Problem]
Currently, Linux network configuration in Neutron heavily relies on shell commands, like ip, brctl, ipset, iptables, etc. Shell commands makes Neutron agents inefficient and really hard to operate in high load environment. In our production deployment scaling from 50 - 500 physical machines per region, 50+ virtual instances per machine, the Neutron agents run extremely slowly and sometimes unresponsive.

There is a blueprint that switch openflow operations from shell to ryu-
based pure python library, but it is not sufficient.

[Solution-1]
I'd like to introduce a pure Python netlink library: pyroute2. It supports network configuration including ip-link, ip-route, ip-netns, tc, ipset and iptables in the roadmap, and is also compatible with python3. It only requires standard library which is also awesome, because you don't need to rely on other unstable third-party libraries that makes dependency hard to maintain. Moreover, it supports transactional local DB operations for network configuration called IPDB.

Doc Link: http://docs.pyroute2.org/general.html
Pypi Link: https://pypi.python.org/pypi/pyroute2

I should first issue a rfe for discussion. Forgot it. :-)
Blueprint Link: https://blueprints.launchpad.net/neutron/+spec/pure-python-linuxnet-conf

[Solution-2]
Currently pyroute2 still doesn't support whole functionality of ipset and iptables, but they are definitely on the roadmap. I'm not sure its progress. I've forked this project and will try to involve in if possible to make sure it evolves as expected. What I suggest is that if possible, should we open a new project, pyosnetconf or networking-linuxnet-conf, whatever, that implements OpenStack's own python library for Linux network configuration. It may be much more aggressive, but still meaningful to neutron.

I'm OK with the two solutions mentioned above. I'd like to get feedback
as much as possible to move forward. Anyway, I strongly suggest to make
it work.

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: rfe

** Description changed:

  [Problem]
- Currently, Linux network configuration in Neutron heavily relies on shell commands, like ip, brctl, ipset, iptables, etc. Shell commands makes Neutron agents inefficient and really hard to operate in high load environment. In our production deployment scaling from 50 - 500 physical machines per region, 50+ virtual instances per machine, the Neutron agents run extremely slowly and sometimes unresponsive. 
+ Currently, Linux network configuration in Neutron heavily relies on shell commands, like ip, brctl, ipset, iptables, etc. Shell commands makes Neutron agents inefficient and really hard to operate in high load environment. In our production deployment scaling from 50 - 500 physical machines per region, 50+ virtual instances per machine, the Neutron agents run extremely slowly and sometimes unresponsive.
  
  There is a blueprint that switch openflow operations from shell to ryu-
- based python library, but it is not sufficient.
+ based pure python library, but it is not sufficient.
  
  [Solution-1]
  I'd like to introduce a pure Python netlink library: pyroute2. It supports network configuration including ip-link, ip-route, ip-netns, tc, ipset and iptables in the roadmap, and is also compatible with python3. It only requires standard library which is also awesome, because you don't need to rely on other unstable third-party libraries that makes dependency hard to maintain. Moreover, it supports transactional local DB operations for network configuration called IPDB.
  
  Doc Link: http://docs.pyroute2.org/general.html
  Pypi Link: https://pypi.python.org/pypi/pyroute2
  
  I should first issue a rfe for discussion. Forgot it. :-)
  Blueprint Link: https://blueprints.launchpad.net/neutron/+spec/pure-python-linuxnet-conf
  
  [Solution-2]
  Currently pyroute2 still doesn't support whole functionality of ipset and iptables, but they are definitely on the roadmap. I'm not sure its progress. I've forked this project and will try to involve in if possible to make sure it evolves as expected. What I suggest is that if possible, should we open a new project, pyosnetconf or networking-linuxnet-conf, whatever, that implements OpenStack's own python library for Linux network configuration. It may be much more aggressive, but still meaningful to neutron.
  
  I'm OK with the two solutions mentioned above. I'd like to get feedback
  as much as possible to move forward. Anyway, I strongly suggest to make
  it work.

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1492714

Title:
  RFE: Pure Python driven Linux network configuration

Status in neutron:
  New

Bug description:
  [Problem]
  Currently, Linux network configuration in Neutron heavily relies on shell commands, like ip, brctl, ipset, iptables, etc. Shell commands makes Neutron agents inefficient and really hard to operate in high load environment. In our production deployment scaling from 50 - 500 physical machines per region, 50+ virtual instances per machine, the Neutron agents run extremely slowly and sometimes unresponsive.

  There is a blueprint that switch openflow operations from shell to
  ryu-based pure python library, but it is not sufficient.

  [Solution-1]
  I'd like to introduce a pure Python netlink library: pyroute2. It supports network configuration including ip-link, ip-route, ip-netns, tc, ipset and iptables in the roadmap, and is also compatible with python3. It only requires standard library which is also awesome, because you don't need to rely on other unstable third-party libraries that makes dependency hard to maintain. Moreover, it supports transactional local DB operations for network configuration called IPDB.

  Doc Link: http://docs.pyroute2.org/general.html
  Pypi Link: https://pypi.python.org/pypi/pyroute2

  I should first issue a rfe for discussion. Forgot it. :-)
  Blueprint Link: https://blueprints.launchpad.net/neutron/+spec/pure-python-linuxnet-conf

  [Solution-2]
  Currently pyroute2 still doesn't support whole functionality of ipset and iptables, but they are definitely on the roadmap. I'm not sure its progress. I've forked this project and will try to involve in if possible to make sure it evolves as expected. What I suggest is that if possible, should we open a new project, pyosnetconf or networking-linuxnet-conf, whatever, that implements OpenStack's own python library for Linux network configuration. It may be much more aggressive, but still meaningful to neutron.

  I'm OK with the two solutions mentioned above. I'd like to get
  feedback as much as possible to move forward. Anyway, I strongly
  suggest to make it work.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1492714/+subscriptions